JP6393495B2 - Image processing apparatus and object recognition method - Google Patents

Description

  The present invention relates to an image processing apparatus and an object recognition method, and more particularly to a technique for recognizing the shape and posture of a certain object from a two-dimensional image in which the object is captured.

  Conventionally, a technique for detecting or recognizing a predetermined object from a two-dimensional image taken by a camera has been widely used. For example, the application range is wide, such as a monitoring camera that tracks a specific person, a driving support system that detects and warns an obstacle around the vehicle, and a gesture input that controls a computer by actions such as gestures and hand gestures. Many of these various applied technologies require detecting the posture of an object in addition to the shape and position of the object.

  2. Description of the Related Art Conventionally, a technique has been proposed in which an image of the shape, position, and posture of a hand is recognized so that a target device can be operated with a hand gesture by hand (see, for example, Patent Document 1). In the information input device described in Patent Document 1, based on the observation data of the environment including the user, the foreground including the user is separated from the background including the environment other than the foreground, and the three-dimensional model is learned. Estimate where the individual foreground model already modeled is located in the environment and its position and orientation.

JP 2013-205983 A

  As in the technique described in Patent Document 1, a target object image extracted from a captured image (hereinafter referred to as a target image) is compared with a plurality of modeled object images (hereinafter referred to as comparative images). Thus, when recognizing the shape and orientation of a target object, a recognition method based on a feature amount is frequently used. In this method, feature amounts extracted by calculation from the target image and the comparison image are compared, and the shape and orientation of the target object are recognized from the comparison image having the closest feature amount to the target image.

  However, when the comparison is performed using only one or several feature amounts calculated from the images, there is a problem that misrecognition increases. For example, when the comparison is performed using the feature amounts of only one to several representative points set on the object, the feature amounts are close to each other if the image similarity with respect to the representative points is high. Even if the degree of similarity is low, the comparatively incorrect image may be selected as the image closest to the target image.

  In particular, when comparing images of various postures are generated for one object and the posture of the target object is recognized by comparing these with the target image, the shapes of the plurality of generated comparative images themselves are similar to each other. It will be. Therefore, in a simple comparison using a feature amount calculated only for a representative point on the object or one feature amount calculated from the overall shape of the object, the image that is originally the closest comparison image of the incorrect solution is the image closest to the target image There was a problem that the correct posture was often not recognized.

  The present invention has been made to solve such a problem, and by comparing the feature amount of an object calculated from each of the target image and a comparison image prepared in advance, the target object captured in the captured image is obtained. The purpose is to make it possible to recognize the shape and posture more correctly.

In order to solve the above-described problems, in the present invention, for a target image obtained by extracting a target object from a captured image and normalizing the size, each pixel position of the boundary of the object and each feature amount at each pixel position are calculated. To do. Then, based on each calculated pixel position and each feature amount and each pixel position and each feature amount calculated in advance in the same manner for a plurality of comparison images, a comparison image with the highest degree of matching is searched. The shape or orientation of the object related to the searched comparison image is specified as the shape or orientation of the object shown in the captured image. Further, in the present invention, when calculating the feature amount of one pixel among the pixels located at the extracted boundary, a binary representing a pattern in which other pixels located at the boundary appear around the one pixel is represented by a bit string. The feature amount is calculated as a feature amount. Further, in the present invention, when searching for a comparative image having the highest degree of coincidence with the target image, the Hamming distance between each of the comparative images and the binary feature amount relating to the pixel of the target image for each pixel position on the extracted boundary The number of pixel positions where it is determined whether or not there is a pixel having a binary feature amount equal to or smaller than a predetermined threshold, and the pixel having a binary feature amount whose Hamming distance is equal to or smaller than the predetermined threshold is present in the vicinity. The comparison image with the largest number is detected as the comparison image with the highest degree of coincidence.

  According to the present invention configured as described above, the feature amount is compared between the target image extracted from the photographed image and the comparison image for each pixel position constituting the boundary of the target object. Then, the comparison image with the highest degree of coincidence is searched as the image closest to the target image, and the shape or posture of the target object is specified by the searched comparison image. As a result, it is possible to perform a comparison with higher accuracy than when comparing based on the feature amount of only the representative point on the object, or comparing based on one feature amount calculated from the overall shape of the object. It is possible to more accurately recognize the shape and posture of the target object shown in the captured image.

It is a block diagram which shows the function structural example of the image processing apparatus by this embodiment. It is a figure for demonstrating the principal component direction analyzed by the principal component analysis part of this embodiment. It is a figure for demonstrating the calculation method of the binary feature-value by this embodiment. It is a figure for demonstrating the example of a process of the image matching part by this embodiment. It is a flowchart which shows the procedure of the process which produces | generates and memorize | stores comparison data in the operation example of the image processing apparatus by this embodiment. 6 is a flowchart illustrating a procedure of processing for recognizing the shape or posture of a target object from a captured image in an operation example of the image processing apparatus according to the present embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating a functional configuration example of the image processing apparatus 100 according to the present embodiment. As shown in FIG. 1, the image processing apparatus 100 according to the present embodiment includes, as its functional configuration, a captured image acquisition unit 11, an object extraction unit 12, a principal component analysis unit 13, a normalization processing unit 14, a boundary extraction unit 15, A feature amount calculation unit 16, a comparison image acquisition unit 17, a comparison data generation unit 18, an image matching unit 19, and a comparison data storage unit 20 are provided.

  Each of the functional blocks 11 to 19 can be configured by any of hardware, DSP (Digital Signal Processor), and software. For example, when configured by software, each of the functional blocks 11 to 19 is actually configured by including a CPU, RAM, ROM, etc. of a computer, and is stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. Is realized by operating.

  The captured image acquisition unit 11 acquires a two-dimensional image generated by capturing a real space using the monocular camera 200. In the example of FIG. 1, a monocular camera 200 is connected to the image processing apparatus 100 such as a personal computer, and the captured image acquisition unit 11 acquires a two-dimensional image captured by the monocular camera 200 in real time. However, the present invention is not limited to this. For example, a two-dimensional image captured by the monocular camera 200 may be stored in a memory, and the captured image acquisition unit 11 may later capture the two-dimensional image stored in the memory.

  The object extraction unit 12 removes the background from the captured image acquired by the captured image acquisition unit 11 and extracts a target image that is an image of the target object. For example, the object extracting unit 12 performs foreground extraction processing on the captured image, thereby extracting an image of the recognition target object that appears in the captured image. A known method can be adopted for the foreground extraction process. For example, foreground extraction processing using graph cut can be applied. Further, the foreground extraction process may be performed using learning data generated from the teacher data using the image of the target object and the background image as the teacher data.

  The principal component analysis unit 13 performs principal component analysis on the object image (the target image extracted by the object extraction unit 12 and the comparison image acquired by the comparison image acquisition unit 17 described later), and the first principal component direction and A second principal component direction is defined. Principal component analysis is a known process for summarizing and expressing information represented by a large number of original explanatory variables into several principal components. In this embodiment, each pixel value of the object image is used as an explanatory variable, and coordinate conversion is performed on these explanatory variables to generate two variables (a first principal component direction and a second principal component direction) that serve as a comprehensive index. . As shown in FIG. 2, the first principal component direction is an approximately long direction of the object, and the second principal component direction is an approximately short direction of the object.

  The normalization processing unit 14 normalizes the object image so that the object image has a normalized size. Specifically, the normalization processing unit 14 normalizes the object image based on the first principal component direction and the second principal component direction specified by the principal component analysis unit 13. The normalization referred to here is a process of enlarging or reducing the size of the object image so that the size of the feature vector with respect to the first principal component direction and the second principal component direction is 1. As a result, when the same target object is photographed, for example, even if the size of the object image shown in the photographed image varies depending on the photographing distance, the object image having the same size can be arranged after the normalization process. it can.

  The boundary extraction unit 15 extracts a boundary (silhouette) of the object from the object image normalized by the normalization processing unit 14. The extraction of the boundary can be performed by, for example, a so-called edge detection process (a process of specifying a location where the brightness or color of the image changes sharply (discontinuously)).

  The feature amount calculation unit 16 calculates each pixel position of the boundary extracted by the boundary extraction unit 15 and each feature amount at each pixel position. If a feature amount is obtained for each pixel position on the boundary, the calculation method is arbitrary. For example, the feature amount calculation unit 16 calculates a binary feature amount described below for each pixel position on the boundary. FIG. 3 is a diagram for explaining a method of calculating a binary feature amount.

  First, as shown in FIG. 3A, the feature amount calculation unit 16 sets the size of one side to n (n is an arbitrary integer equal to or greater than 2) with the pixel 31 of interest in the silhouette image of the object as the center. n × n voxels 32 are set. The example of FIG. 3A shows a case where the voxel 32 is set with n = 5. The target pixel 31 is one of the pixels having the boundary line 33 of the object in the silhouette image.

  Next, the feature quantity calculation unit 16 encodes each voxel 32 as shown in FIG. Specifically, the feature amount calculation unit 16 sets the value to 1 for voxels 32 including pixels of the boundary line 33 and sets the value to 0 for voxels 32 not including the boundary line 33. The encoding of 0 and 1 may be a reverse pattern. That is, the value of voxel 32 including the pixel of the boundary line 33 may be 0, and the value of voxel 32 not including the pixel of boundary line 33 may be 1.

  Further, as shown in FIG. 3C, the feature amount calculation unit 16 obtains the values of each encoded voxel 32 in four directions of the horizontal direction, the vertical direction, and the two diagonal directions, and arranges them in order. To generate a 20-dimensional binary string. This is the binary feature value to be obtained. It should be noted that the four directions listed here and the order of arrangement thereof are merely examples, and the present invention is not limited to these. The feature amount calculation unit 16 moves the target pixel 31 one by one along the boundary line 33 of the object, and calculates a binary feature amount for each pixel position on the boundary line 33.

  The comparative image acquisition unit 17 acquires the same image as the target object to be recognized from the captured image as a comparative image. For example, the comparative image acquisition unit 17 acquires a CG image of an object generated by a personal computer or the like. The form of acquisition is arbitrary. For example, a personal computer is connected to the image processing apparatus 100, and the comparative image acquisition unit 17 acquires a comparative image directly from the personal computer. Alternatively, a comparison image generated by a personal computer or the like may be stored in a memory, and the comparison image acquisition unit 17 may capture the comparison image stored in the memory.

  When it is desired to recognize the shape of the target object from the captured image, the comparative image acquisition unit 17 acquires comparative images relating to various objects having different shapes. When it is desired to recognize the posture of the target object (the direction in which the target object is facing) from the captured image, the comparative image acquisition unit 17 acquires various comparative images having different postures with respect to the same object. When it is desired to recognize both the shape and posture of the target object from the captured image, the comparative image acquisition unit 17 acquires various comparative images having different postures for various objects having different shapes. Here, the comparative image acquisition unit 17 acquires various comparative images together with information indicating at least one of the shape and posture of the object represented by the comparative image.

  The above-described principal component analysis unit 13, normalization processing unit 14, boundary extraction unit 15, and feature amount calculation unit 16 perform the same processing on the comparison image acquired by the comparison image acquisition unit 17. This is performed in advance before the target object to be recognized is photographed by the monocular camera 200. In this case, the principal component analysis unit 13 performs a principal component analysis on the comparison image acquired by the comparison image acquisition unit 17 to determine a first principal component direction and a second principal component direction.

  The normalization processing unit 14 normalizes the comparison image so that the comparison image acquired by the comparison image acquisition unit 17 has a normalized size. Thereby, the size of the object image represented by the normalized comparison image and the size of the object image extracted from the captured image and normalized can be adjusted to substantially the same size. The boundary extraction unit 15 extracts the boundary (silhouette) of the object from the comparison image normalized by the normalization processing unit 14. The feature amount calculation unit 16 calculates each pixel position of the boundary extracted by the boundary extraction unit 15 and each binary feature amount at each pixel position.

  The comparison data generation unit 18 performs pixel processing of each pixel position generated in advance by performing each process of the principal component analysis unit 13, the normalization processing unit 14, the boundary extraction unit 15, and the feature amount calculation unit 16 on a plurality of comparison images related to the object. The set of binary feature values is combined with information indicating at least one of the shape and orientation of the object represented by the comparison image for each of the plurality of comparison images to generate comparison data. Then, the generated comparison data is stored in the comparison data storage unit 20.

  The image matching unit 19 includes each pixel position and each binary feature amount calculated for the target image by the feature amount calculation unit 16, each pixel position and each of the plurality of comparison images stored in advance in the comparison data storage unit 20. Based on the binary feature amount, a comparative image having the highest degree of matching is searched. Then, the shape or orientation of the object related to the searched comparison image is specified as the shape or orientation of the target object shown in the captured image.

  Specifically, the image matching unit 19 determines the difference between each pixel position calculated for the target image by the feature amount calculation unit 16 and each pixel position calculated for the comparison image, and the target image by the feature amount calculation unit 16. On the basis of the difference between each binary feature amount calculated with respect to the comparison image and each binary feature amount calculated with respect to the comparison image, the comparison image having the smallest overall difference in binary feature amount at each pixel position is Search as a comparative image with a high degree of coincidence.

  FIG. 4 is a diagram for explaining a processing example of the image matching unit 19. FIG. 4A shows a silhouette image of the target object extracted from the captured image. 4B to 4D show silhouette images of comparison objects extracted from a plurality of comparison images.

  First, as shown in FIG. 4A, the image matching unit 19 includes the pixel position (x1, y1) calculated for the target image by the feature amount calculation unit 16 and the binary feature amount P11 calculated at the pixel position. And get. Next, the image matching unit 19 targets a plurality of comparative images shown in FIGS. 4B to 4D as pixel positions within a predetermined distance (range indicated by reference numeral 41) from the pixel position (x1, y1). Then, it is searched whether there is a comparison image having a binary feature amount in which the Hamming distance with the binary feature amount P11 is equal to or less than a predetermined threshold.

  The Hamming distance is a value obtained by comparing the corresponding bits of the 20-bit binary feature amount calculated from each of the target image and the comparison image and counting the number of bits indicating different values. When a comparison image in which the Hamming distance is equal to or less than a predetermined threshold is searched, the image matching unit 19 adds a score to the comparison image.

  The comparison image shown in FIG. 4B is substantially the same shape and size as the target image in FIG. 4A (the size is normalized), and within a predetermined distance from the pixel position (x1, y1). The pixel feature has a binary feature amount whose Hamming distance from the binary feature amount P11 is equal to or less than a predetermined threshold value. Therefore, a score is added to this comparison image. On the other hand, since the comparison images shown in FIGS. 4C and 4D are different in shape from the target image in FIG. 4A, the binary feature is located at a pixel position within a predetermined distance from the pixel position (x1, y1). There is no binary feature amount whose Hamming distance to the amount P11 is equal to or less than a predetermined threshold. Therefore, no score is added to these comparative images.

  The image matching unit 19 sequentially performs the above processing for each pixel position on the boundary line of the target object shown in FIG. As a result, the comparative image having the highest score is extracted as the comparative image having the highest degree of coincidence with the target image. Then, the shape or posture of the object related to the extracted comparison image (stored in the comparison data storage unit 20) is specified as the shape or posture of the target object shown in the captured image.

  5 and 6 are flowcharts showing an operation example of the image processing apparatus 100 according to the present embodiment configured as described above. FIG. 5 shows a procedure of processing for generating and storing comparison data in advance. FIG. 6 shows a processing procedure for recognizing the shape or posture of the target object from the captured image.

  In FIG. 5, first, the comparative image acquisition unit 17 acquires a CG image of an object generated by a personal computer or the like as a comparative image (step S1). For example, the comparison image acquisition unit 17 acquires various comparison images having different postures with respect to various objects having different shapes together with information indicating the shape and posture of the object represented by the comparison images. Next, the principal component analysis unit 13 performs a principal component analysis on the comparison image acquired by the comparison image acquisition unit 17, and determines a first principal component direction and a second principal component direction (step S2).

  Further, the normalization processing unit 14 normalizes the comparison image so that the comparison image acquired by the comparison image acquisition unit 17 has a normalized size (step S3). Subsequently, the boundary extraction unit 15 extracts the boundary (silhouette) of the object from the comparison image normalized by the normalization processing unit 14 (step S4). Then, the feature amount calculation unit 16 calculates each pixel position of the boundary extracted by the boundary extraction unit 15 and each binary feature amount at each pixel position (step S5).

  Finally, the comparison data generation unit 18 represents the set of each pixel position and each binary feature amount generated for the plurality of comparison images by the feature amount calculation unit 16 for each of the plurality of comparison images. Is stored in the comparison data storage unit 20 together with information indicating the shape and posture (step S6). Thus, the preparation for recognizing the shape and posture of the target object from the captured image is completed.

  In FIG. 6, the captured image acquisition unit 11 acquires a two-dimensional image generated by capturing a real space using the monocular camera 200 (step S11). In addition, the object extraction unit 12 removes the background from the captured image acquired by the captured image acquisition unit 11 and extracts a target image of the object (step S12). Next, the principal component analysis unit 13 performs a principal component analysis on the target image extracted by the object extraction unit 12, and determines a first principal component direction and a second principal component direction (step S13).

  Further, the normalization processing unit 14 normalizes the target image so that the target image extracted by the object extraction unit 11 has a normalized size (step S14). Further, the boundary extraction unit 15 extracts the boundary (silhouette) of the object from the target image normalized by the normalization processing unit 14 (step S15). Subsequently, the feature amount calculation unit 16 calculates each pixel position of the boundary extracted by the boundary extraction unit 15 and each feature amount at each pixel position (step S16).

  Finally, the image matching unit 19 stores each pixel position and each binary feature amount calculated for the target image by the feature amount calculation unit 16 and each pixel stored in advance in the comparison data storage unit 20 for each of the plurality of comparison images. Based on the position and each binary feature amount, a comparison image having the highest degree of coincidence is searched, and the shape or posture of the object related to the searched comparison image is specified as the shape or posture of the target object in the captured image (step) S17).

  As described above in detail, in the present embodiment, each pixel position of the boundary and each feature amount at each pixel position are calculated for the target image obtained by extracting the target object from the captured image and normalizing the size. Then, based on each calculated pixel position and each feature amount and each pixel position and each feature amount calculated in advance in the same manner for a plurality of comparison images, a comparison image with the highest degree of matching is searched. Then, the shape or orientation of the object related to the searched comparison image is specified as the shape or orientation of the target object shown in the captured image.

  According to the present embodiment configured as described above, the feature amount is compared between the target image of the object extracted from the photographed image and the comparison image for each pixel position constituting the boundary of the target object. . Then, the comparison image with the highest degree of coincidence is searched as the image closest to the target image, and the shape or posture of the object is specified by the searched comparison image. As a result, the comparison is based on the feature value of only the representative point on the object as in the past, and the comparison is more accurate than the comparison based on the single feature value calculated from the overall shape of the object. And the shape and posture of the target object appearing in the captured image can be recognized more correctly.

  In the above embodiment, the example in which the principal component analysis is performed to identify the first principal component direction and the second principal component direction and the object image is normalized by these two directions has been described. It is not limited. As long as the object image can be normalized, the method based on principal component analysis is not necessarily required.

  Moreover, although the said embodiment demonstrated the example which calculates a Hamming distance in image matching, this invention is not limited to this. Any method can be applied to the present invention as long as it can calculate the degree of similarity between the feature amount calculated from the target image and the feature amount calculated from the comparison image.

  In the above embodiment, the principal component analysis unit 13 and the normalization processing unit 14 of the same image processing apparatus 100 perform the processing for calculating the feature amount of the object from the target image and the processing for extracting the feature amount of the object from the comparison image. Although the example performed using the boundary extraction unit 15 and the feature amount calculation unit 16 has been described, the present invention is not limited to this. For example, the process of extracting the feature amount of the object from the comparison image is performed by a personal computer or the like different from the image processing apparatus 100, and the comparison data obtained as a result is stored in the comparison data storage unit 20 in advance. Also good.

  In addition, each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the gist or the main features thereof.

DESCRIPTION OF SYMBOLS 11 Captured image acquisition part 12 Object extraction part 13 Principal component analysis part 14 Normalization process part 15 Boundary extraction part 16 Feature-value calculation part 17 Comparison image acquisition part 18 Comparison data generation part 19 Image matching part 20 Comparison data storage part 100 Image processing Device 200 Monocular camera

Claims (2)

An object extraction unit that extracts a target image that is an image of the target object by removing the background from the captured image;
A normalization processing unit that normalizes the object image so that the object image has a normalized size;
A boundary extraction unit that extracts the boundary of the object from the object image normalized by the normalization processing unit;
A feature amount calculation unit that calculates each pixel position of the boundary extracted by the boundary extraction unit and each feature amount at each pixel position;
A set of each pixel position and each feature value generated in advance by performing the same processing as the normalization processing unit, the boundary extraction unit, and the feature value calculation unit for the plurality of comparison images related to the object is compared with the plurality of comparison images. A comparison data storage unit that stores, for each image, information indicating at least one of the shape and orientation of the object represented by the comparison image;
Based on each pixel position and each feature amount calculated for the target image by the feature amount calculation unit, and each pixel position and each feature amount stored for each of the plurality of comparison images in the comparison data storage unit, An image matching unit that searches for a comparative image having the highest degree of coincidence and identifies the shape or orientation of the object related to the searched comparative image as the shape or orientation of the object in the captured image ;
The feature amount calculation unit
Among the pixels located at the boundary extracted by the boundary extraction unit, when calculating the feature value of one pixel, a binary representing a pattern in which other pixels located at the boundary appear around the one pixel by a bit string Calculate the feature value as a feature value,
The image matching unit
When searching for a comparison image having the highest degree of coincidence with the target image, for each comparison image, for each pixel position on the boundary extracted by the boundary extraction unit, a binary feature amount related to the pixel of the target image It is determined whether or not there is a pixel having a binary feature amount whose Hamming distance is equal to or smaller than a predetermined threshold, and the pixel position where it is determined that a pixel having a binary feature amount whose Hamming distance is equal to or smaller than a predetermined threshold exists in the vicinity An image processing apparatus that detects a comparison image having the largest number of images as a comparison image having the highest degree of coincidence . A first step in which an object extraction unit of the image processing apparatus extracts a target image that is an image of a target object by removing the background from the captured image;
A second step in which the normalization processing unit of the image processing apparatus normalizes the object image so that the object image extracted by the object extraction unit has a normalized size;
A third step in which the boundary extraction unit of the image processing device extracts the boundary of the object from the object image normalized by the normalization processing unit;
A fourth step in which the feature amount calculation unit of the image processing apparatus calculates each pixel position of the boundary extracted by the boundary extraction unit and each feature amount at each pixel position;
The image matching unit of the image processing apparatus includes each pixel position and each feature amount calculated for the target image by the feature amount calculation unit, and each pixel stored in advance in the comparison data storage unit for each of a plurality of comparison images. A comparison image with the highest degree of matching is searched based on the position and each feature amount, and the shape or posture of the object related to the searched comparison image is specified as the shape or posture of the object shown in the captured image ,
In the fourth step, when calculating the feature amount of one pixel among the pixels located on the boundary extracted by the boundary extraction unit, the feature amount calculation unit sets the boundary around the one pixel. A binary feature value representing a pattern in which other pixels appear in a bit string is calculated as a feature value,
In the fifth step, when the image matching unit searches for a comparison image having the highest degree of coincidence with the target image, for each comparison image, for each pixel position on the boundary extracted by the boundary extraction unit. , It is determined whether there is a pixel having a binary feature amount whose Hamming distance with a binary feature amount relating to the pixel of the target image is equal to or less than a predetermined threshold, and a binary feature amount whose Hamming distance is equal to or less than a predetermined threshold value An object recognition method, comprising: detecting a comparison image having the largest number of pixel positions determined to be present in the vicinity as a comparison image having the highest degree of matching .

Images