JP6369131B2 - Object recognition apparatus and object recognition method - Google Patents

Description

  The present invention relates to an object recognition apparatus and an object recognition method for recognizing the position of an object, and particularly for recognizing a placement position of an object when an automatic guided vehicle transports an object such as a pallet or a case. The present invention relates to an object recognition apparatus and an object recognition method. In particular, the present invention relates to an object recognition apparatus that recognizes a specific image of a predetermined object such as a palette, not an unspecified image such as a human face.

Various object recognition apparatuses that recognize an object based on image information obtained by photographing the object have been proposed.
For example, Patent Document 1 discloses a device that generates a three-dimensional image including distance information based on two image data captured by a pair of cameras provided at different positions, and performs object recognition based on the three-dimensional image. Is disclosed.

Further, Patent Document 2 uses a distance image (three-dimensional image) obtained by a TOF (Time Of Flight) camera to calculate a feature amount by making a histogram of feature points, and recognizes an object by comparison with a reference pattern. An apparatus for performing is disclosed.
Further, Patent Document 3 discloses an apparatus for recognizing an object by calculating the three-dimensional coordinates of the object from the two-dimensional image of the object and the distance image, and applying the data of the three-dimensional model as a reference pattern. ing.

JP 09-218041 A International Publication No. 2010/140613 Pamphlet JP 2012-123781 A

  When the object is recognized from the two-dimensional image, the object can be recognized based on the luminance value pattern of each pixel. In this case, if the shape of the object to be recognized is a simple pattern, erroneous recognition is likely to occur, and erroneous recognition also occurs depending on illumination conditions. Furthermore, it is difficult to determine the actual size of the pattern on the two-dimensional image. In particular, when the shooting direction of the camera is oblique, it is difficult to accurately recognize the pattern on the two-dimensional image.

For this reason, in the object recognition apparatus of patent documents 1-3, the feature point of an object is extracted from the three-dimensional image which image | photographed the object, and the object is recognized. However, it is necessary to extract a large number of feature point candidates for pattern matching, which increases the calculation load. In addition, when the image information includes a feature point included in the background or an object having a feature point similar to the target object, there is a risk of erroneous recognition. In addition, erroneous recognition may occur even when the object is smaller than the entire image.
As described in Patent Document 3, when image features of an object extracted from a two-dimensional image are calculated using a three-dimensional image and collated with three-dimensional model data, there are many The calculation load when extracting the feature points becomes large. In order to reduce such a calculation load, it is conceivable to reduce the number of feature point candidates for performing pattern matching. However, in this case, there is a problem that the accuracy of pattern matching is lowered.

  An object of the present invention is to reduce a calculation load in an object recognition apparatus without reducing the accuracy of object recognition.

Hereinafter, a plurality of modes will be described as means for solving the problems. These aspects can be arbitrarily combined as necessary.
An object recognition apparatus according to an aspect of the present invention includes a model storage unit, a three-dimensional image input unit, an image feature point detection unit, a feature amount calculation unit, and a pattern matching unit. The model storage unit stores model data representing a geometric feature of an object defined by a set of points on the object surface to be recognized and a set of line segments formed by connecting the points. The three-dimensional image input unit inputs a three-dimensional image including an object. The image feature point detection unit detects feature points in the three-dimensional image. The feature amount calculation unit calculates a feature amount at each point on the three-dimensional image. The pattern matching unit compares the similarity between the feature points detected by the image feature point detection unit and the model data based on the feature amounts between the feature points and the interpolation points satisfying a specific geometric condition. The object is recognized.
Depending on the shape of the object, an attribute as a point representing the feature of the object can be given to a point that is not a feature point on the model. By making such a point an interpolation point that satisfies a specific geometric condition with the feature point, it is possible to downsample the feature point when performing pattern matching, and as a result, other feature points existing in the background It is possible to identify and reliably recognize the object. Thereby, the calculation load of the pattern matching process can be reduced.

The pattern matching unit may extract a predetermined number of feature points satisfying a predetermined geometric condition as matching candidate points from the feature points detected by the image feature point detection unit.
By setting the geometric conditions of the feature points representing the features of the object, it is easy to downsample the matching candidate points for matching from the feature points detected from the three-dimensional image. Can be reduced.

The interpolation point may be an intermediate point between one or more pairs of feature points.
In the part where the edge of the object is intermittent, there is a point that is not a feature point on the object between the detected feature points. Such a point can be effective information when recognizing an object present on the image. For this reason, matching accuracy can be improved by using an intermediate point of the feature points as an interpolation point as a matching candidate point.

The pattern matching unit may recognize an object based on a histogram of matching candidate points created by the feature amount calculation unit.
For calculating the feature amount, a known technique can be used. For example, a normal direction histogram, a distance histogram, a luminance level histogram, and the like can be used. More specifically, FPFH (Fast Point Feature Histograms) can be used.
The object may be a pallet having three feature points. In this case, the predetermined number of feature points satisfying a predetermined geometric condition are three feature points whose angles formed by the three feature points are within a predetermined range.

The three-dimensional image input unit may include a TOF (Time Of Flight) camera or a stereo camera.
In this case, a distance image of an object photographed using a TOF camera or a stereo camera can be input as a three-dimensional image. As a technique for detecting a feature point using a distance image photographed by a TOF camera and a stereo camera, a known technique can be used.

The image feature point detection unit may detect feature points by performing Harris processing or SIFT (Scale Invariant Feature Transform) processing. Also, ISS (Intrinsic Shape Signatures) may be used.
The feature point detection method only needs to be able to detect a point at which a color or luminance change increases for each pixel of the image, and a known technique can be used in addition to the method using the Harris operator or SIFT operator.

The object recognition device includes a two-dimensional image input unit that inputs a two-dimensional image including an object photographed from substantially the same viewpoint as the three-dimensional image, and a two-dimensional image processing unit that detects an image feature of the object in the two-dimensional image. Furthermore, you may provide.
The image feature point detection unit may detect the feature point only in a region corresponding to the image feature of the object detected by the two-dimensional image processing unit in the three-dimensional image.
In this case, since the feature points are detected in the region corresponding to the image features of the object specified by the two-dimensional image, the calculation load is reduced compared to the extraction of the feature points from the normal three-dimensional image. Can be speeded up.
The feature amount calculation unit may calculate a feature amount at a feature point in the region in the three-dimensional image, and the pattern matching unit may use the feature amount of the feature point in the region in the three-dimensional image.

An object recognition method according to another aspect of the present invention includes the following steps.
◎ Step of saving model data representing geometric features of an object defined by a set of points on the object surface to be recognized and a set of line segments connecting the points ◎ 3D image containing the object ◎ Step of detecting feature point in 3D image ◎ Step of calculating feature amount between feature point and interpolation point satisfying specific geometric condition for feature point ◎ Model based on feature amount In this case, by executing a program including each step by a microprocessor including a CPU, a ROM, and a RAM, the object is detected from a feature point with a small number of samplings. Recognition can be performed, and as a result, the calculation load can be reduced.

  ADVANTAGE OF THE INVENTION According to this invention, the object recognition apparatus which can reduce a calculation load can be provided, without reducing the accuracy of object recognition.

Explanatory drawing of the unmanned forklift which mounts an object recognition apparatus. The block diagram which shows schematic structure of an object recognition apparatus. The flowchart which shows schematic structure of an object recognition process. The block diagram which shows an example of a three-dimensional image input part. The characteristic figure which shows an example of scale (sigma) and DoG filter output value. Explanatory drawing of the feature point detected using the SIFT feature-value. Explanatory drawing regarding the detection method of a Harris feature point. Explanatory drawing of the feature point detected using the Harris operator. The flowchart of an image feature-value detection process-pattern matching process. Explanatory drawing which shows the geometric conditions between feature points. Explanatory drawing of the matching candidate point extracted by applying geometric conditions. Explanatory drawing which shows the relationship between a feature point and an interpolation point. Explanatory drawing of the feature-value calculation method by FPFH. Explanatory drawing of the feature-value calculation method by FPFH. The graph which shows FPFH feature-value. Explanatory drawing of the FPFH feature-value of a matching candidate point.

(1) Schematic Configuration FIG. 1 is an explanatory diagram of an unmanned forklift equipped with an object recognition device.
The unmanned forklift 2 includes a main body 21 that can autonomously travel based on the control of a control unit (not shown) installed therein.
The front surface of the main body 21 is provided with a pair of forks 23 and 24 arranged to be movable up and down, and the pallet 3 can be lifted up and conveyed.
Furthermore, the main body 21 includes an object recognition device 22. The object recognition device 22 includes a camera (details will be described later) that can acquire at least image information in front of the main body 21. The object recognition device 22 recognizes the palette 3 that is a recognition target from a three-dimensional image based on image information captured by the camera. You may provide the camera of the object recognition apparatus 22 in the front-end | tip part of the forks 23 and 23. FIG.

The pallet 3 is provided with insertion ports 31 and 32 into which the forks 23 and 24 can be inserted, for example, as illustrated. On the surface of the pallet 3 where the insertion ports 31 and 32 are formed, vertical surfaces 33, 34 and 35 sandwiching the insertion ports 31 and 32 are formed.
The object recognition device 22 can recognize and identify the pallet 3 including its position, shape and size by detecting feature points of the vertical surfaces 33 to 35 formed on the surface facing the unmanned forklift 2.

The unmanned forklift 2 transports, for example, a pallet 3 on which articles are placed in a warehouse. The control unit of the unmanned forklift 2 grasps its own current position in the environment in the warehouse, determines the travel route with the vicinity of the pallet 3 to be transported as the target position, and performs travel control. The object recognition device 22 recognizes and identifies the position, shape, and size of the pallet 3 based on the image direction taken by the camera at the target position. The control unit of the unmanned forklift 2 inserts the forks 23 and 24 into the insertion ports 31 and 32 of the pallet 3 to lift the pallet 3 and move it to the next target position.
There are cases where an article is placed on the upper surface of the pallet 3 and cases where no article is placed, and the object recognition device 22 makes it possible to recognize the pallet 3 in either case.

FIG. 2 is a block diagram illustrating a schematic configuration of the object recognition apparatus.
The object recognition device 22 includes a three-dimensional image input unit 41, an image feature point detection unit 42, a feature amount calculation unit 43, a pattern matching unit 44, and a model storage unit 45.
The three-dimensional image input unit 41 inputs a three-dimensional image of an object. For example, the three-dimensional image input unit 41 may include a stereo camera having two cameras at different positions. In this case, distance data is displayed on feature points on the two-dimensional image due to the parallax between the two two-dimensional images. The processed distance image is input as a three-dimensional image. In addition, a distance image generated based on distance data obtained by a TOF camera or a distance sensor can be input as a three-dimensional image.

  The image feature point detector 42 detects feature points for the three-dimensional image. Specifically, the image feature point detection unit 42 can detect a feature point by detecting a point where the color or shading changes drastically for each pixel of the three-dimensional image. From this, the image feature point detection unit 42 can detect the feature points by, for example, a method using a Harris operator. In addition to this, the image feature point detection unit 42 can detect a feature point by a method using a SIFT operator, and can detect a feature point using a known technique.

  The feature amount calculation unit 43 calculates the feature amount at each point on the three-dimensional image. In particular, the feature amount calculation unit 43 calculates feature amounts for matching candidate points when pattern matching is performed. The feature amount can be calculated by a normal direction histogram generated based on an angle formed by a normal direction vector at each point and a normal vector of a peripheral feature point. Specifically, FPFH (Fast Point Feature Histograms) can be used. Further, since the input image information is a three-dimensional image, the feature amount calculation unit 43 can create a distance histogram at each point based on the distance data. In addition to this, it is possible to create a luminance histogram based on the luminance of each pixel and calculate the feature amount of each point.

The pattern matching unit 44 extracts matching candidate points on the three-dimensional image based on the feature points detected by the image feature point detection unit 42, and calculates the feature amount of the extracted matching candidate points and the model data. The object is recognized by comparing the similarities.
For example, the pattern matching unit 44 extracts one or a plurality of feature point pairs having a specific geometric relationship from the feature points detected by the image feature point detection unit 42. For example, when the vertical surfaces 33 to 35 of the pallet 3 have a plurality of feature points at the edges in contact with the floor surface, the feature points located on the vertical surfaces 33 to 35 are considered to be substantially on a straight line. Therefore, a line segment connecting the feature point located at the center of the three feature points and the other two feature points forms an angle of approximately 180 degrees. If the pallet 3 conforms to the standard, the distance between the three feature points falls within a predetermined range.
Therefore, the pattern matching unit 44 sets the feature points of these three points as matching candidate points if they are separated by a distance within a predetermined range and the angle formed by the line segments connecting the points is within the predetermined range. Furthermore, the pattern matching unit 44 sets a middle point between the feature point located at the center and the other two feature points as a pattern matching candidate point. In this way, the pattern matching unit 44 down-samples the feature points by selecting the matching candidate points for performing pattern matching.

Next, the pattern matching unit 44 compares the feature amount calculated by the feature amount calculation unit 43 with the model data stored in the model storage unit 45 for each of the matching candidate points. In this way, object recognition is performed.
The model storage unit 45 stores two-dimensional model data representing the geometric characteristics of the object. The data of the two-dimensional model is used when calculating the position and orientation of an object when recognizing an object photographed in a three-dimensional image, and is composed of, for example, a set of points and points. Defined by a set of line segments. Specifically, the basic model data includes 1) a distance range between adjacent points of 3 feature points, 2) a distance range between points at both ends of 3 points, and 3) a distance between 3 feature points. Angle range to be formed 4) Average and variance for each dimension (angle) of the FPFH histogram of each feature point / interpolation point.
In addition, the data of the two-dimensional model can be a feature point obtained from a three-dimensional image of an object and a feature amount at an intermediate point between the feature points. In this case, the feature point detection method is the same method as the image feature point detection unit 42, and the feature amount calculation method of each point is the same method as the feature amount calculation unit 43.
The model storage unit 45 includes a predetermined area of an EEPROM, RAM, hard disk, or other storage device.

(2) Schematic Configuration of Processing FIG. 3 is a flowchart showing a schematic configuration of object recognition processing.
The object recognition apparatus 22 performs object recognition from image information by a control unit configured by a microprocessor including a CPU, a ROM, a RAM, and the like executing a predetermined program.
In step S301, the control unit stores data of the two-dimensional model. The control unit stores, in the model storage unit 45, data of a two-dimensional model defined by a set of points on the object surface to be recognized and a set of line segments formed by connecting the points.

  In step S302, the three-dimensional image input unit 41 inputs a three-dimensional image of an object. As described above, the three-dimensional image input unit 41 can input a three-dimensional image including distance information captured by the stereo camera 51.

In step S303, the image feature point detector 42 detects feature points in the three-dimensional image. Specifically, the image feature point detection unit 42 detects feature points on a three-dimensional image using a Harris operator. As described above, the feature points need only be detected at each pixel of the three-dimensional image as a feature point at which the color or shading changes, and can be detected using a SIFT operator or other known techniques. Is possible.
The pattern matching unit 44 extracts a predetermined number of feature points that satisfy a predetermined geometric condition as matching candidate points, and further determines an interpolation point that satisfies a specific geometric condition for the extracted feature points.

In step S304, the feature amount calculation unit 43 calculates feature amounts at the detected feature points and interpolation points.
The feature amount calculation unit 43 creates a normal direction histogram by FPFH for the matching candidate points, and calculates the feature amount at each point.

  In step S <b> 305, the pattern matching unit 44 performs pattern matching by obtaining a similarity between a group of matching candidate points configured by feature points and interpolation points and data of the two-dimensional model. The pattern matching unit 44 extracts feature points that satisfy the geometric condition and compares the extracted interpolation points related to the extracted feature points with the data of the two-dimensional model as matching candidate points. Therefore, the calculation load in pattern matching can be reduced and the calculation speed can be improved.

(3) Three-dimensional image input A specific example of the three-dimensional image input unit 41 will be described below.
FIG. 4 is a block diagram illustrating an example of a three-dimensional image input unit.
The three-dimensional image input unit 41 illustrated in FIG. 4 generates a three-dimensional image having distance information based on the image information from the stereo camera 51.

In the stereo camera 51, two cameras that take a two-dimensional image are arranged at different positions. The stereo camera 51 is attached to the front surface of the unmanned forklift 2, for example, and takes at least a two-dimensional image of the front positions of the forks 23 and 24.
Parameters relating to the position and orientation of the two cameras with respect to the unmanned forklift 2 are set in advance and stored in a predetermined storage area of the object recognition apparatus 1.

The frame memory 52 stores two two-dimensional image information photographed by the stereo camera 51. The two-dimensional image information stored in the frame memory 52 is provided to the two-dimensional feature point extraction unit 54 via the image bus 53.
The two-dimensional feature point extraction unit 54 detects a two-dimensional feature point based on luminance information for each of the two two-dimensional image information. When a predetermined pattern is formed on the vertical surfaces 33 to 35 of the pallet 3, two-dimensional feature points can be detected along the predetermined pattern. In this case, the two-dimensional feature point extraction unit 54 can track two pieces of two-dimensional image information in the same direction, detect a turning point on the outer periphery of a predetermined pattern, and use this as a two-dimensional feature point. .

  The association processing unit 55 generates a candidate for the correspondence relationship between the two-dimensional feature points extracted by the two-dimensional feature point extracting unit 54. The association processing unit 55 generates a transformation matrix for each correspondence relationship candidate, obtains the conversion error, and extracts a correspondence relationship candidate that minimizes the conversion error.

  The coordinate calculation unit 56 calculates the coordinates of each feature point of the correspondence candidate extracted by the association processing unit 55. For example, since the two two-dimensional image information obtained by the stereo camera 51 is obtained by photographing the same point with cameras having different positions, it is possible to obtain distance information to the feature point. The coordinate calculation unit 56 generates three-dimensional coordinates of each feature point based on the feature points of the two two-dimensional image information associated by the association processing unit 55, and obtains the three-dimensional image information including the distance information. Generate.

Image information captured by a TOF (Time Of Flight) camera can be used as the three-dimensional image. A TOF camera measures the distance to an object by irradiating infrared light with LEDs provided around the camera and measuring the time until reflected light reflected from the object is observed by the camera. . The TOF camera can measure the distance of each pixel constituting the image, and generates a distance image displayed with different luminances according to the distance of each pixel from the camera.
When the above-described stereo camera 51 is used, it is necessary to perform processing for associating two two-dimensional images. However, by using image information captured by the TOF camera, the processing cost is reduced and the processing time is reduced. Real-time processing is possible by reducing

(4) Image Feature Point Detection A known method can be used as a method for detecting a feature point from input three-dimensional image information in the image feature point detection unit 42.
For example, in a feature point detection method using a SIFT operator, an extreme value of a scale whose shape changes rapidly when enlargement or reduction is performed is used.

  In the feature point detection method using the SIFT operator, when a LoG (Laplacian of Gaussian) filter of various scales is applied, the feature point has an extreme value suitable for use as a feature point. Actually, instead of the LoG filter, the image is smoothed using a Gaussian function with various standard deviations, and a DoG (Difference of Gaussian) filter that calculates the difference is approximated to the LoG filter.

FIG. 5 is a characteristic diagram showing an example of the scale σ and the DoG filter output value.
FIG. 5B is a characteristic diagram in which the scale σ is changed and the output value of the DoG filter is detected for image information on which a predetermined pattern is formed as shown in FIG.
In the illustrated example, it can be seen that the extreme value of the scale σ at which the output value of the DoG filter is the largest is about 10.15.
The image feature point detector 42 determines the candidate position of the feature point on the image information in FIG. 5A and the scale σ obtained in FIG.
Thus, the image feature point detection unit 42 can select a feature point having an invariant feature amount with respect to a scale change.

FIG. 6 is an explanatory diagram of feature points detected using SIFT feature values.
In FIG. 6, many characteristic points appear on the front surface 61 of the pallet 3 including the vertical surfaces 33 to 35 and the insertion ports 31 to 32. In particular, as shown in the first region 62 and the second region 63, the pallet 3 is an edge portion in contact with the floor surface, and many feature points appear in the vicinity of the insertion ports 31 to 32.
As a result of changing the scale or rotating the image and calculating the SIFT feature value, many feature points appear in the edge portion where the pallet 3 is in contact with the floor surface in the same manner.

As an image feature point detection method, a method using a Harris operator is known.
In the feature point detection method using the Harris operator, a point at which the sum of squares (SSD: Sum of Squared Difference) of luminance value differences increases when an image in a predetermined region is slightly shifted is detected.
FIG. 7 is an explanatory diagram relating to a Harris feature point detection method.
Consider a case where the predetermined area 72 set on the image 70 is slightly shifted on the assumption that the object image 71 exists on the image 70.
As shown in FIG. 7A, when the predetermined area 72 is set on the object image 71, the change in the luminance value due to the slight shift of the image 70 is small. Therefore, in this case, it is difficult to detect feature points regardless of the direction of movement.

As shown in FIG. 7B, when the predetermined area 72 is set on the edge of the object image 71, the luminance value changes little with respect to the vertical movement of the figure, but the right and left of the figure With respect to movement in the direction, the change in luminance value becomes large. Therefore, it is possible to detect the edge in the vertical direction as a feature point by moving the image 70 in the left-right direction in the figure.
Further, as shown in FIG. 7C, when the predetermined area 72 is set near the corner of the object image 71, the change in the luminance value becomes large regardless of the direction in which the image 70 is moved. Therefore, the corner position of the recognition target object can be detected as the feature point.

  According to the feature point detection method using such a Harris operator, when the object 74 is placed on the floor surface 73, as shown in FIG. It can be detected as a feature point. For example, the predetermined area 76 including the corner 75 can be slightly shifted, and the feature point can be detected based on the sum of squares of the luminance values.

FIG. 8 is an explanatory diagram of feature points detected using a Harris operator.
In FIG. 8, many feature points appear on the front surface 81 of the pallet 3 including the vertical surfaces 33 to 35 and the insertion ports 31 to 32. In particular, as shown in the first region 82, the second region 83, and the third region 84, many feature points appear at the edge portions where the vertical surfaces 33 to 35 of the pallet 3 are in contact with the floor surface.
As a result of calculating the Harris feature amount when the scale is changed or by rotating the image, many feature points appear in the edge portions where the vertical surfaces 33 to 35 are in contact with the floor surface in the same manner.
Therefore, if a group of feature points satisfying a predetermined geometric condition is extracted from the feature points detected from the three-dimensional image information as matching candidate points, pattern matching can be performed efficiently.

(5) Image feature point detection to pattern matching process The pattern matching unit 44 compares the feature amount of the matching candidate point with the feature amount of the data of the two-dimensional model based on the feature point detected by the image feature point detection unit 42. Then, the object on the image is recognized and identified. This image feature amount detection process to pattern matching process will be described in detail below.
FIG. 9 is a flowchart of image feature amount detection processing to pattern matching processing. FIG. 9 shows processing after the three-dimensional image information is input, and shows processing corresponding to steps S303 to S305 in FIG.
In step S901, the image feature point detector 42 detects a feature point from a point group on the three-dimensional image. Specifically, the image feature point detection unit 42 extracts feature points on the three-dimensional image using a feature point detection method using a Harris operator.

  In step S902, the pattern matching unit 44 extracts matching candidate points based on geometric conditions. For example, when extracting three feature points corresponding to the vertical surfaces 33 to 35 positioned in front of the pallet 3, the distance between the feature points is a distance that matches a predetermined standard, and the feature points are connected. A geometric condition is set such that the angle of the line segment is within a predetermined range.

FIG. 10 is an explanatory diagram showing geometric conditions between feature points.
As described above, when the feature points are detected using the Harris operator on the three-dimensional image information obtained by photographing the palette 3, the three points located at the lower front of the palette 3 can be extracted as the feature points.
The three feature points appear corresponding to the edges where the vertical surfaces 33 to 35 of the pallet 3 are in contact with the floor surface. Therefore, the three feature points appear side by side on a substantially straight line.
From this, three features such that the line segment connecting the first feature point and the second feature point and the line segment connecting the first feature point and the third feature point are in an angle range close to 180 degrees. By extracting the points, feature points corresponding to the front shape of the pallet 3 can be extracted.

As shown in FIG. 10, it is assumed that the feature point located at the center among the three feature points is the first feature point 91, and the second feature point 92 and the third feature point 93 are on both sides thereof.
The distance α1 between the first feature point 91 and the second feature point 92, the distance α2 between the first feature point 91 and the third feature point 93, and the distance between the second feature point 92 and the third feature point 93 are β. . Further, an angle formed by a line segment 94 connecting the first feature point 91 and the second feature point 92 and a line segment 95 connecting the first feature point 91 and the third feature point 93 is defined as θ.

When the size of the pallet 3 conforms to the standard, the values of the distances α1, α2, and β between the first feature point 91 to the third feature point 93 are within a predetermined range. For example, the size of a T11 type pallet that is often used in Japan is 1100 × 1100 × 144 (mm). Therefore, it is considered that the values of the distances α1 and α2 are about 500 (mm) and the value of the distance β is about 1000 (mm).
From this, it is possible to set the geometric conditions such that the distances α1 and α2 are in the range of 340 (mm) to 600 (mm) and the distance β is in the range of 800 (mm) to 1050 (mm).

Further, since the first feature point 91 to the third feature point 93 are substantially on a straight line, the angle θ is approximately 180 degrees. Therefore, the geometric condition that the value of cos θ is −0.95 or less can be set.
By setting such a geometric condition, three feature points for pattern matching with the reference pattern of the palette 3 can be stably extracted.

FIG. 11 is an explanatory diagram of matching candidate points extracted by applying a geometric condition.
By applying such a geometric condition, matching candidate points 101 to 103 corresponding to the lower edge of the vertical surfaces 33 to 35 of the pallet 3 can be extracted as shown in FIG.
However, in the first area 105 and the second area 106 where the pallet 3 does not exist, matching candidates having similar characteristics exist.
In order to eliminate matching candidates existing in the first region 105 and the second region 106, interpolation points that satisfy a specific geometric condition are set for the feature points.
In step S903, the pattern matching unit 44 determines an interpolation point that satisfies a specific geometric condition for the feature point based on the feature point extracted as a matching candidate.

FIG. 12 is an explanatory diagram showing the relationship between feature points and interpolation points.
In FIG. 12, an image obtained by photographing the palette 3 in FIG. 1 is shown as a palette 111, and images corresponding to the vertical surfaces 33 to 35 are shown as vertical surfaces 112 to 114.
In the image shown in FIG. 12, the pallet 111 is placed on the floor surface 110 and has vertical surfaces 112, 113, 114 on the front surface of the pallet 111. There are a first region 115, a second region 116, and a third region 117 corresponding to the bent surface formed by the vertical surfaces 112, 113, and 114 and a part of the floor surface 110. The feature points 118, 119, and 120 appear at positions where the vertical surfaces 112, 113, and 114 and the floor surface 110 are in contact with each other in the first region 115, the second region 116, and the third region 117.

In the middle of the first area 115 and the second area 116 and in the middle of the second area 116 and the third area 117, the fourth area 121 and the fifth area corresponding to the plane continuous from the floor surface 110 to the inside of the pallet 111, respectively. Region 122 exists. Interpolation points 123 and 124 are set in the fourth area 121 and the fifth area 122, respectively.
The control unit extracts the feature points 118, 119, and 120 and the interpolation points 123 and 124 as matching candidate points.

  In step S904, the feature amount calculation unit 43 calculates a feature amount for each of the matching candidate points. The feature amount can be calculated from a normal direction histogram generated for each matching candidate point based on an angle between the normal direction vector and the normal vector of the peripheral feature points. Specifically, FPFH (Fast Point Feature Histograms) can be used.

13 and 14 are explanatory diagrams of a feature amount calculation method using FPFH.
As shown in FIG. 13, the neighboring points Pk1 to Pk5 and P6 to P11 are set for the target feature point Ps.
As shown in FIG. 14, angle differences α, φ, and θ with respect to the three-dimensional coordinates u, v, and w between the normal line ns of the feature point Ps and the normal line nt of the neighboring point Pt are obtained.
Three histograms obtained by dividing 11 degrees of 180 degrees by the angle differences α, φ, θ between the normal line ns of the feature point Ps and the normal lines nt of 11 neighboring points Pt are arranged to form a 33-dimensional histogram.

FIG. 15 is a graph showing the FPFH feature amount.
When the feature amount by FPFH is graphed as described above, the frequency (%) appears on the vertical axis for the angle differences α, φ, θ as shown in FIG.
In this way, the control unit calculates the FPFH feature quantity at the feature points 118, 119, 120 extracted as matching candidate points and the interpolation points 123, 124.

In step S905, the control unit reads the data of the two-dimensional model stored in the model storage unit 45.
When performing pattern matching using the FPFH feature value, the FPFH feature value is calculated in advance based on a captured image of the object that is the recognition target, and is stored in the model storage unit 45 as data of a two-dimensional model.

  The calculation of the FPFH feature amount is performed from, for example, a predetermined number of images obtained by photographing the pallet 3 under various conditions in which the distance and angle are changed. Furthermore, an average value and a variance value of the calculated FPFH feature values are calculated and used as a feature point model. When the average value of the FPFH feature value is 33 and the variance value is 33, and a total of 66 values are used as a model of one feature point, the three feature points 118, 119, and 120 located in front of the pallet 3 198 values are stored in the model storage unit 45 as features of the palette 3.

Similarly, for the interpolation points 123 and 124 that are intermediate points between the feature points 118, 119, and 120, the FPFH feature amount is calculated and stored in the model storage unit 45.
FIG. 16 is an explanatory diagram of the FPFH feature amount of the matching candidate point.
FIG. 16A shows the FPFH feature quantity of the feature point 119 located at the center of the front surface of the pallet 3. Since the feature point 119 is located at the center of the front surface of the pallet 3, the angle difference in the normal direction from the neighboring points is stable.
FIG. 16B shows the FPFH feature values of the feature points 118 and 120 located on the left and right of the front surface of the pallet 3. The feature points 118 and 120 are affected by the feature points on the side surface of the pallet 3, and the angle difference in the normal direction from the neighboring points is not stable. Is getting bigger.
FIG. 16C shows the FPFH feature values of the interpolation points 123 and 124 located on a plane having no corners, and the variation in the central graph corresponding to the angle difference φ is small.

In step S905, the pattern matching unit 44 reads out the FPFH feature amounts of the three feature points and the two interpolation points stored in the model storage unit 45.
In step S906, the pattern matching unit 44 collates the feature amount of the matching candidate point with the feature amount of the two-dimensional model. Specifically, the difference between the read FPFH feature quantity of the two-dimensional model and the FPFH feature quantity of the matching candidate point calculated by the feature quantity calculation unit 43 is calculated, and the group of matching candidate points having the smallest difference is calculated. Select.

  In step S907, the pattern matching unit 44 outputs a recognition result (article position). Specifically, for the group of matching candidate points determined to be most similar to the data of the two-dimensional model, the position of the object is calculated using the three-dimensional coordinates of each feature point.

With the procedure as described above, the position of the image of the palette 3 photographed in the three-dimensional image can be recognized. In this case, since the matching candidate points for performing the pattern matching are extracted by the feature points that satisfy a predetermined geometric condition, it is possible to reduce the calculation load.
In addition, since the interpolation points satisfying a predetermined geometric condition are set as the matching candidate points for the feature points, the accuracy of pattern matching can be improved.

(6) Features of Embodiment The object recognition device (for example, the object recognition device 22) includes a model storage unit (for example, model storage unit 45), a three-dimensional image input unit (for example, three-dimensional image input unit 41), An image feature point detection unit (for example, image feature point detection unit 42), a feature amount calculation unit (for example, feature amount calculation unit 43), and a pattern matching unit (for example, pattern matching unit 44) are provided. The model storage unit (model storage unit 45) determines the geometric features of an object (for example, palette 3) defined by a set of points on the surface of the object to be recognized and a set of line segments connecting the points. Save the model data that you want to represent. The 3D image input unit (3D image input unit 41) inputs a 3D image including an object. The image feature point detector (image feature point detector 42) detects feature points (for example, feature points 118, 119, 120) in the three-dimensional image. The feature amount calculation unit (feature amount calculation unit 43) calculates a feature amount (for example, FPFH feature amount) at each point on the three-dimensional image. The pattern matching unit (pattern matching unit 44) includes feature points (feature points 118, 119, 120) detected by the image feature point detection unit (image feature point detection unit 42) and feature points (feature points 118, 119, 120) based on the feature quantity at the interpolation points (interpolation points 123 and 124) satisfying a specific geometric condition (intermediate point of the feature points) with respect to the model data, Recognize.
Depending on the shape of the object, a point representing the feature of the object may appear at a point that is not a feature point on the model. By making such a point an interpolation point that satisfies a specific geometric condition with the feature point, it is possible to downsample the feature point when performing pattern matching, and as a result, other feature points existing in the background It is possible to identify and reliably recognize the object. Thereby, the calculation load of the pattern matching process can be reduced.

(7) Other Embodiments Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications can be made without departing from the scope of the invention. In particular, a plurality of embodiments and modifications described in this specification can be arbitrarily combined as necessary.

In the embodiment, the image feature point detection unit 42 detects feature points in all the regions of the obtained three-dimensional image. However, based on the two-dimensional image obtained at the same time, the region of the three-dimensional image may be limited in advance. For example, any one of the stereo cameras 51 functions as a two-dimensional image input unit that inputs a two-dimensional image including the palette 3 photographed from substantially the same viewpoint as the three-dimensional image. In addition, a two-dimensional image processing unit (not shown) extracts the image features of the palette in the two-dimensional image. This extraction operation is performed at high speed. Subsequently, the two-dimensional image processing unit converts the camera coordinates (horizontal direction and vertical direction) of the extracted palette into real-world coordinates with the camera position as the origin by calibration.
The image feature point detection unit 42 detects a feature point in the region corresponding to the image feature of the palette narrowed down by the two-dimensional image processing unit with respect to the three-dimensional image. In this case, since the feature point is detected in the region corresponding to the image feature of the object specified by the two-dimensional image, the calculation load can be reduced compared to the extraction of the feature point from the normal three-dimensional image, High speed is possible.
The feature amount calculation unit 43 calculates feature amounts at feature points in the region in the three-dimensional image. The pattern matching unit 44 uses feature amounts of feature points in the region in the three-dimensional image.
Note that the camera that acquires the two-dimensional image may be provided separately from the camera that acquires the three-dimensional image.

  The interpolation point may be any point as long as it satisfies a specific geometric condition for at least two feature points, and may not be an intermediate point between the two feature points. For example, the orientation of SIFT or the main direction of ISS may be used as those extreme values, inflection points, or one point of a line segment intersecting the line segment in the direction of two feature points.

  The present invention can be applied to a recognition device that recognizes an object from a three-dimensional image.

2 Unmanned Forklift 3 Pallet 21 Main Body 22 Object Recognition Device 31 Insertion Port 32 Insertion Port 33 Vertical Surface 34 Vertical Surface 35 Vertical Surface 41 Three-dimensional Image Input Unit 42 Image Feature Point Detection Unit 43 Feature Quantity Calculation Unit 44 Pattern Matching Unit 45 Model Storage unit 51 Stereo camera 52 Frame memory 53 Image bus 54 Two-dimensional feature point detection unit 55 Association processing unit 56 Coordinate calculation unit 61 Front surface 62 First region 63 Second region 70 Image 71 Object image 72 Predetermined region 73 Floor surface 74 Object 75 Corner 76 Predetermined region 81 Front surface 82 First region 83 Second region 84 Third region 91 First feature point 92 Second feature point 93 Third feature point 110 Floor surface 111 Pallet 112 Vertical surface 113 Vertical surface 114 Vertical surface 115 First 1 area 116 2nd area 117 3rd area 118 Feature point 119 Feature point 120 Feature points 121 the fourth area 122 fifth region 123 interpolation points 124 interpolation points

Claims (12)

A model storage unit that stores model data representing geometric features of the object defined by a set of points on the surface of the object to be recognized and a set of line segments connecting the points;
A three-dimensional image input unit for inputting a three-dimensional image including the object;
An image feature point detector for detecting feature points in the three-dimensional image;
A feature amount calculation unit for calculating a feature amount at each point on the three-dimensional image;
The feature points detected by the image feature point detection unit and interpolation points that satisfy a specific geometric condition for the feature points are grouped as matching candidate points for pattern matching, and matching candidates included in each group A pattern matching unit that recognizes the object by comparing the similarity with the data of the model based on the feature amount of the point;
An object recognition apparatus comprising: The object recognition apparatus according to claim 1, wherein the interpolation point is an intermediate point between one or more pairs of feature points. The object according to claim 1, wherein the pattern matching unit extracts a predetermined number of feature points satisfying a predetermined geometric condition as the matching candidate points from the feature points detected by the image feature point detection unit. Recognition device.   The object recognition apparatus according to claim 1, wherein the pattern matching unit recognizes the object based on a histogram created by the feature amount calculation unit for each of the matching candidate points.   The object recognition apparatus according to claim 4, wherein the feature amount calculation unit creates a histogram described by FPFH (First Point Feature History). The object is a pallet having three feature points;
The object recognition apparatus according to claim 3 , wherein the predetermined number of feature points satisfying the predetermined geometric condition are three feature points whose angles formed by the feature points of the three feature points are within a predetermined range. .   The object recognition apparatus according to claim 1, wherein the three-dimensional image input unit includes a TOF (Time Of Flight) camera or a stereo camera.   The object recognition apparatus according to claim 1, wherein the image feature point detection unit detects the feature points by performing Harris processing or SIFT (Scale Invariant Feature Transform) processing. A two-dimensional image input unit for inputting a two-dimensional image including the object photographed from substantially the same viewpoint as the three-dimensional image;
A two-dimensional image processing unit for detecting an image feature of the object in the two-dimensional image;
The object recognition apparatus according to claim 1, further comprising:   The image feature point detection unit detects the feature point only in a region corresponding to the image feature of the object detected by the two-dimensional image processing unit in the three-dimensional image. The object recognition apparatus described. The feature amount calculation unit calculates a feature amount at the feature point in the region in the three-dimensional image,
The object recognition apparatus according to claim 10, wherein the pattern matching unit uses the feature amount of the feature point in the region in the three-dimensional image. Storing model data representing geometric features of the object defined by a set of points on the object surface to be recognized and a set of line segments connecting the points;
Inputting a three-dimensional image including the object;
Detecting feature points in the three-dimensional image;
Calculating a feature amount at the feature point and an interpolation point that satisfies a specific geometric condition for the feature point;
Recognizing an object by comparing the similarity with the model data based on the feature amount; and
An object recognition method comprising:

Images