JP2008015895A - Image processor and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor and image processing program for accurately determining the adequacy of the result of deducing the attitude of an object to be recognized. <P>SOLUTION: When the attitude of an object to be recognized is deduced, first of all, the attitude of the object is deduced, on the basis of a contour of the object extracted from a photographic image of the object. Then, the three-dimensional shape of the object is deduced, on the basis of the result of attitude deduction. The three-dimensional shape of the object is detected, and the object is restored in three dimensions. By collating the three-dimensional shape deduced, based on the result of the attitude deduction with the result of the three-dimensional restoration restored based on the result of the three-dimensional detection, validity of the result of the attitude deduction is decided. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing program for estimating the posture of an object from a captured image of the object.

As this type of image processing apparatus, for example, as described in Non-Patent Document 1 below, an apparatus that estimates the posture using the contour of an object is known.
Hendrik PA Lensch, et al. “A Silhouette-Based Algorithm for Texture Registration and Stitching” ”, Graphical Models 63.2001, pp.245-262.

  However, as in Non-Patent Document 1, when the posture of an object is estimated using the outline of the object, which is simply two-dimensional information, erroneous recognition of the object or incorrect posture estimation may occur.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus and an image processing program that can accurately determine the validity of an object posture estimation result.

  An image processing apparatus according to the present invention estimates a posture of an object based on a contour of an object extracted from a captured image of the object, and estimates a three-dimensional shape of the object based on a posture estimation result obtained by the posture estimation unit. Validity of the posture estimation result by the posture estimation means by collating the 3D estimation means, the detection means for detecting the three-dimensional shape of the object, and the three-dimensional shape estimated by the 3D estimation means and the detection result by the detection means And determining means for determining.

  If the posture of the object is simply estimated based on the contour of the object, the estimation error may be large as a region occupied by the object in three dimensions even if the estimation error in two dimensions is small. In the present invention, the three-dimensional shape of the object is estimated using the posture estimation result estimated based on the contour of the object, and the estimated three-dimensional shape is collated with the detection result of the three-dimensional shape of the object obtained separately. Thus, the validity of the object posture estimation result is determined three-dimensionally. Therefore, it is possible to accurately determine the validity of the object posture estimation result.

  Preferably, the determination unit determines the validity of the posture estimation result by comparing the three-dimensional shape estimated by the 3D estimation unit and the detection result with respect to a preset comparison target region in the object. In this case, the validity of the posture estimation result of the object can be determined more accurately by presetting a region having characteristics suitable for the collation target and collating the region.

  Preferably, the image processing apparatus further includes selection means for selecting an area having a degree of overlap lower than a predetermined value as an area to be collated in an image obtained by superimposing a plurality of images obtained by imaging an object from a plurality of viewpoints. A region where the overlapping degree is lower than a predetermined value in an image obtained by superimposing a plurality of images obtained by imaging an object from a plurality of viewpoints is a characteristic region in the object. Therefore, by selecting a characteristic area as a comparison target area, it is possible to sufficiently adequately determine the validity of the object posture estimation result.

  An image processing program according to the present invention estimates a posture of an object based on a contour of an object extracted from a captured image of the object, and estimates a three-dimensional shape of the object based on a posture estimation result in the posture estimation step Validity of the posture estimation result in the posture estimation step by collating the 3D estimation step, the detection step for detecting the three-dimensional shape of the object, and the three-dimensional shape estimated in the 3D estimation step with the detection result in the detection step And a determination step of determining

  If the posture of the object is simply estimated based on the contour of the object, the estimation error may be large as a region occupied by the object in three dimensions even if the estimation error in two dimensions is small. In the present invention, by estimating the three-dimensional shape of the object using the posture estimation result estimated based on the contour of the object, and comparing the estimated three-dimensional shape with the detection result of the three-dimensional shape of the object, The validity of the pose estimation result is determined three-dimensionally. Therefore, it is possible to accurately determine the validity of the object posture estimation result.

  Preferably, in the determination step, the validity of the posture estimation result is determined by collating the three-dimensional shape estimated in the 3D estimation step with the detection result for the collation target region estimated in advance in the object. In this case, the validity of the posture estimation result of the object can be determined more accurately by presetting a characteristic region suitable for the collation target and collating the region.

  Preferably, the computer further executes a selection step of selecting an area whose overlapping degree is lower than a predetermined value in an image obtained by superimposing a plurality of images obtained by imaging an object from a plurality of viewpoints. A region where the overlapping degree is lower than a predetermined value in an image obtained by superimposing a plurality of images obtained by imaging an object from a plurality of viewpoints is a characteristic region in the object. Therefore, by selecting a characteristic area as a comparison target area, it is possible to sufficiently adequately determine the validity of the object posture estimation result.

  According to the present invention, it is possible to accurately determine the validity of an object posture estimation result. Thereby, the estimation accuracy of posture estimation can be improved.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of an image processing apparatus and an image processing program according to the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an embodiment of an image processing apparatus according to the present invention. The image processing apparatus 1 of this embodiment is mounted on a robot (not shown) that grips a container with a handle such as a teapot or a mug as a recognition target object.

  In FIG. 1, an image processing apparatus 1 inputs cameras 2A and 2B that capture a recognition target object and images captured by these cameras 2A and 2B, performs predetermined image processing, and estimates the posture of the recognition target object. An image processing unit 3, a monitor unit 4 for displaying the processing result of the image processing unit 3, and a data storage unit 5 for accumulating and storing a database used for image processing by the image processing unit 3 are provided.

  The cameras 2A and 2B are, for example, CCD cameras, and are provided in both eyes (not shown) of the robot. That is, the cameras 2A and 2B are arranged so as to capture an image of an object from two different viewpoints.

  The image processing unit 3 may be configured as dedicated hardware specialized for object recognition processing, or a general-purpose computer such as a personal computer is used to cause the computer to execute an image processing program as software. May be. At this time, the image processing program is provided by, for example, a storage medium such as a CD-ROM, a DVD, or a ROM, or a semiconductor memory. The image processing program may be provided via a network as a computer data signal superimposed on a carrier wave.

  Further, the processing result of the image processing unit 3 is sent to the grip control processing unit 6. The grip control processing unit 6 controls a robot hand (not shown) so as to grip the recognition target object based on the posture of the recognition target object estimated by the image processing unit 3.

  FIG. 2 is a flowchart showing an outline of a processing procedure by the image processing unit 3. First, captured images of recognition target objects by the cameras 2A and 2B are acquired (step 11). An example of images captured by the cameras 2A and 2B is shown in FIG. FIG. 3A shows an outline of an image (left image) captured by the camera 2A arranged in the left eye part of the robot, and FIG. 3B shows an image taken by the camera 2B arranged in the right eye part of the robot. An outline of an image (right image) is shown. In the present embodiment, as an example, the recognition target object is a teapot.

  In the left image (FIG. 3A), a teapot image 31L is shown as an image showing the teapot of the posture as viewed from the left eye of the robot, and a milk pack image 32L is also shown as an image showing the milk pack. In the right image (FIG. 3B), a teapot image 31R is shown as an image showing the teapot of the posture viewed from the right eye of the robot, and a milk pack image 32R is also shown as an image showing the milk pack.

  Subsequently, the left image and the right image acquired in step 11 are divided into regions for each object whose density values can be regarded as uniform (step 12). In parallel with the region dividing process in step 12, a three-dimensional image is restored from the two-dimensional left image and right image (step 13). This three-dimensional image restoration is performed by, for example, using the concept of binocular parallax by calculating the depth of a point from the position coordinates of the point in the left and right images and the distance between the cameras 2A and 2B. Do.

  Subsequently, based on the two-dimensional image divided in step 12 and the three-dimensional image restored in step 13, the contour of the object is extracted by edge detection or the like (step 14). In the present embodiment, a contour indicating a region of the same color as the teapot color that is the recognition target object is extracted from each of the left and right images.

  Subsequently, the feature amount of the contour of each image obtained in step 14 is extracted (step 15). As the feature amount of the contour, an invariant that is invariant with respect to the position, rotation, and size of the contour is used. For example, in a captured image, there are cases where a plurality of tangent lines parallel to each other can be drawn. In this case, the distance between the tangents is invariant to the position, rotation and size of the contour. Therefore, here, the distance between tangents for each gradient angle is used as an invariant.

  Subsequently, by matching the feature amount of the contour of each image extracted in step 15 with the feature matching data stored as a database in the data storage unit 5, the similarity to the feature matching data of each recognition target object is obtained. Calculate (step 16). Data relating to the posture of the recognition target object corresponding to a plurality of viewpoints is registered as the feature matching data. More specifically, data indicating the contour of the recognition target object corresponding to a plurality of viewpoints and feature amount data extracted from the contour are registered.

  In step 16, it is recognized that the object corresponding to the data with the highest calculated similarity corresponds to the object indicated by the contour of each image. In the present embodiment, it is recognized that the outline of each image indicates a teapot. As a matching method, for example, DP (Dynamic Programming) matching or the like is adopted.

  Subsequently, the reliability of the contour of the left image and the contour of the right image is evaluated, and an image including the contour having the higher reliability of the left and right images is selected (step 17). The reliability is for predicting the reliability of the estimation when estimating the posture of the object using the contour. Specifically, contour complexity and feature similarity are used as the reliability.

  The complexity is expressed by quantifying the complexity of the contour, and specifically, the total number of invariant dimensions for each gradient angle in the contour. In general, a contour having a high degree of complexity has more information on the posture of the recognition target object, and is therefore more reliable in estimating the posture. That is, by evaluating the reliability using the complexity, it is possible to select an image including a highly reliable contour when estimating the posture. However, when the complexity of the contours is the same in the left and right images, an image having a higher feature amount similarity is selected.

  Subsequently, the maximum similarity between the feature amount of the contour of the selected image and the feature amount of each contour in the feature matching data of the type of the recognition target object identified in step 16 (teapot in this embodiment) is obtained. Calculate (step 18). Then, the contour shape corresponding to the maximum similarity in the feature matching data is set as the initial posture of the recognition target object (step 19).

  Subsequently, the posture of the recognition target object is roughly estimated using the outline of the reference image (step 20). That is, the posture is estimated using an image obtained by one camera (monocular posture estimation). The rough estimation is an estimation of the posture that is performed more coarsely than the detailed estimation that is performed later. In step 20, a DT (Distance Transforms) image is created for the contour in the reference image and matched with the contour shape data stored in the data storage unit 5 to perform rotation / translation calculation to determine the posture of the recognition target object. Coarse estimation.

  Subsequently, the validity of the rough posture estimation is determined (step 21). This validity is determined, for example, from the degree of overlap when the contour obtained by posture estimation and the contour extracted from the left and right images are superimposed.

  If it is determined in step 21 that the posture estimation is not correct, a new initial posture is set, and the coarse posture estimation of the recognition target object is performed again. For example, the contour shape corresponding to the highest similarity after the maximum similarity calculated in step 18 is set as a new initial posture of the recognition target object (step 22). And said step 20 is performed again.

  Subsequently, the posture of the recognition target object is estimated in detail using both the left image and the right image (step 22). In the detailed posture estimation process in step 22, a DT image is created for the contours in the left image and the right image, and the object to be recognized is subjected to rotation / translation calculation by matching with the contour shape data stored in the data storage unit 5 Is estimated in more detail than the above rough posture estimation (compound eye posture estimation).

  For example, in the left image, the posture of the recognition target object is estimated as a contour 33L as shown in FIG. 4A, and in the right image, the posture of the recognition target object is a contour as shown in FIG. 4B. Estimated as 33R. Through the processes of steps 11 to 22 described above, the contour of the recognition target object is extracted from the captured image of the recognition target object, and the posture of the recognition object is estimated based on the extracted contour.

  Subsequently, the validity of the posture estimation result is determined (step 23). If it is determined in step 23 that the posture estimation result is not correct, the processing in step 25 described above is executed. If it is determined in step 23 that the posture estimation result is correct, the posture estimation result is sent to the grip control processing unit 6 and displayed on the monitor unit 4 (step 24). In this way, the posture of the recognition target object is estimated by the image processing unit 3.

  Subsequently, the determination of the validity of the posture estimation result in step 23 will be described in more detail. The validity of this detailed posture estimation is determined using the overlapping area data registered in the data storage unit 5. First, the overlapping area data will be described, and a method for registering the overlapping area data as a database in the data storage unit 5 will be described. FIG. 5 is a flowchart showing a procedure for registering data in the data storage unit 5.

  Projected images of the recognition target object are acquired from a plurality of viewpoints, and the sum of luminance values in the projected images of the respective viewpoints is calculated (step 41). For example, as shown in FIG. 6, projection images of the teapot 34 are obtained from a plurality of viewpoints W on the spherical surface U of the virtual sphere assumed to be centered on the teapot 34. The projected image is an image that shows the area occupied by the teapot 34 and the other area by the difference in luminance value. For example, in the projected image shown in FIG. 7, the brightness value of the area outside the contour 35 indicated by the closed curve is low, and the brightness value of the area inside the contour 35 (area indicating the teapot 34) is high.

  By obtaining projected images of the teapot 34 from a plurality of viewpoints W on the spherical surface U, a two-dimensional image showing a plurality of postures of the teapot 34 can be acquired. Thereafter, all the acquired projection images are overlaid and the luminance value is added. FIG. 8 shows an example of the superimposed projected image. Outlines 36 to 38 show the outlines of the teapots in the three projected images.

  Next, as shown in FIG. 8, a region S in which the sum of luminance values is equal to or greater than a threshold is extracted from the two-dimensional image obtained by superimposing the projection images (step 42). This area S is an area occupied by the teapot 34 in the projection images obtained from all the viewpoints W. This region S is approximated by a circle C (step 43). Circle C is set to include region S. The approximate shape is not limited to a perfect circle such as the circle C, but may be an ellipse, a rectangle, or a shape obtained by expanding the region S.

  Next, as shown in FIG. 9, a sphere V on which a circle C is projected for all viewpoints W is calculated (step 44). That is, the sphere V has the same shape in any two-dimensional image obtained from any viewpoint W. The region that protrudes from the sphere V in the recognition target object located at the center of the spherical surface U is a region in which the degree of overlap is lower than a predetermined value in an image obtained by superimposing a plurality of images obtained from a plurality of viewpoints. This area is an area where the shape of the recognition target object is characteristic. For example, the region 35 a that protrudes from the sphere V is a handle of the teapot 34. A region 35b protruding from the sphere V is a spout of a teapot.

  Subsequently, the calculated information of the sphere V is registered in the data storage unit 5 as overlapping area data (step 45). As described above, the overlapping area data of the recognition target object is calculated and registered in the data storage unit 5 in advance before the recognition target object posture estimation.

  Subsequently, the procedure for determining the validity of the posture estimation result in step 23 will be described. FIG. 10 is a flowchart illustrating a procedure for determining the validity of the posture estimation result.

  First, the above-mentioned sphere V registered in the data storage unit 5 is projected on the projection image showing the posture estimation result obtained in step 22 shown in FIG. 2 (step 51). Specifically, as shown in FIG. 11, a sphere V is projected onto a contour 61 indicating the posture estimation result.

  Next, a contour T indicating a region other than the region of the sphere V in the projection image indicating the posture estimation result is obtained (step 52). Specifically, the hatched portion surrounded by the contour T in FIG. 11 indicates a region other than the region of the sphere V in the projection image indicating the posture estimation result. That is, the area indicated by the contour T is selected as the verification target area. Such collation target regions (contours T) are obtained for the posture estimation results in the left and right images, respectively.

  Subsequently, the three-dimensional shape of the region indicated by the contour T is estimated using the left and right images (step 53). For example, using the concept of binocular parallax, a three-dimensional shape is estimated by calculating the depth of a point from the position coordinates in the left and right images of a point and the distance between the cameras 2A and 2B. . This three-dimensional shape (referred to as a three-dimensional estimated shape) is set to a shape larger than the result calculated using the contour T, for example, taking into account errors in posture estimation and errors in converting to a three-dimensional shape Is done.

  On the other hand, the three-dimensional shape of the recognition target object is separately detected, and the recognition target object is three-dimensionally restored (step 54). For example, the three-dimensional shape of the recognition target object is detected by performing three-dimensional measurement using a stereo range finder. Note that the detection of the three-dimensional shape is not limited to being performed using a stereo range finder, and for example, the three-dimensional restored image acquired in step 13 in FIG. 2 may be used.

  Subsequently, the three-dimensional estimated shape obtained in step 53 and the three-dimensional restoration result obtained in step 54 are overlapped (step 55). For example, as shown in FIG. 12, a teapot three-dimensional restoration result 62 is set on three-dimensional coordinates, and is superimposed on the three-dimensional coordinates to set a three-dimensional estimated shape 63 of the region indicated by the contour T.

  Subsequently, a region where the three-dimensional estimated shape on the three-dimensional coordinates overlaps with the three-dimensional restoration result is calculated (step 56). For example, the number of overlapping Voxels is calculated. Further, an area where the 3D estimated shape and the 3D restoration result overlap may be calculated from a 2D screen showing the 3D estimated shape set on the 3D coordinates and the 3D restoration result.

  If the calculated value indicating the size of the overlapping area is equal to or greater than the threshold, it is determined that the posture estimation result is valid (step 57). If the calculated value indicating the size of the overlapping area is smaller than the threshold value, it is determined that the posture estimation result is not valid (step 57), and the process proceeds to step 25 shown in FIG. As described above, the validity of the posture estimation result is determined by collating the three-dimensional estimated shape obtained in step 53 with the detection result obtained in step 54.

  In the above, steps 15 to 22 shown in FIG. 2 constitute posture estimation means (posture estimation step) for estimating the posture of the recognition target object based on the outline of the recognition target object extracted from the captured image of the recognition target object. Steps 51 and 52 shown in FIG. 10 are selection means for selecting, as a collation target area, an area having a degree of overlap lower than a predetermined value in an image obtained by superimposing a plurality of images obtained by imaging a recognition target object from a plurality of viewpoints. Selection step). Step 53 shown in FIG. 10 constitutes 3D estimation means (3D estimation step) for estimating the three-dimensional shape of the recognition target object based on the posture estimation result by the posture estimation means. Step 54 in FIG. 10 constitutes detection means (detection step) for detecting the three-dimensional shape of the recognition target object. Steps 55 to 57 in FIG. 10 are judgment means (judgment step) for judging the validity of the posture estimation result by the posture estimation means by collating the three-dimensional shape estimated by the 3D estimation means with the detection result by the detection means. ).

  By the way, in order to determine the validity of the posture estimation result, it is conceivable to make a determination based on the deviation between the contour or the pattern of the recognition target object extracted from the image captured by the camera and the contour or the pattern of the posture estimation result. However, there are cases where the outline and the arrangement of the patterns are similar even if the postures are different. In this case, even if the difference between the contours or the difference in the arrangement of the patterns is small, the posture estimation result and the actual posture of the recognition target object are greatly deviated. Therefore, it is difficult to accurately determine the validity of the posture estimation result simply by using two-dimensional information.

  On the other hand, in this embodiment, using the posture estimation result estimated based on the contour of the recognition target object, the three-dimensional estimated shape 63 of the recognition target object is set on the three-dimensional coordinates, and the recognition target object Using the three-dimensional detection result, the three-dimensional restoration result 62 of the recognition target object is set on the three-dimensional coordinates. That is, the three-dimensional restoration result 62 and the three-dimensional estimated shape 63 are superimposed on the three-dimensional coordinates. Subsequently, a region where the three-dimensional restoration result 62 and the three-dimensional estimated shape 63 overlap is calculated, and the validity of the posture estimation result of the recognition target object is determined from the result. Therefore, the validity of the posture estimation result of the recognition target object is determined based on the three-dimensional information, and the validity of the posture estimation result can be accurately determined. At this time, it is preferable to superimpose the three-dimensional restoration result and the three-dimensional estimated shape on the three-dimensional coordinates for the characteristic region (the handle in this case) of the recognition target object. Thus, by accurately determining the validity of the posture estimation result, posture estimation can be performed again when the posture estimation result is low in validity, and erroneous recognition of posture estimation can be reliably suppressed.

  Although the above embodiment is applied to a robot that grips an object, the image processing apparatus and the image processing program of the present invention can be applied to other apparatuses and systems that recognize an object and estimate the posture of the object. Applicable.

1 is a block diagram showing a configuration of an embodiment of an image processing apparatus according to the present invention. It is a flowchart which shows the outline of the process sequence by the image process part shown in FIG. It is a figure which shows an example of the left image and right image which were imaged with the two cameras shown in FIG. It is a figure which shows an example of a compound eye attitude | position estimation result. It is a flowchart which shows the procedure which registers data in the data storage part shown in FIG. It is an image figure which shows the viewpoint position of a camera. It is a figure which shows the projection image obtained from a certain viewpoint. It is a figure which shows the image which added the projection image of several viewpoint positions. It is an image figure which shows the sphere V calculated by step 44 of FIG. It is a flowchart which shows the procedure which judges the validity of an attitude | position estimation result. It is a figure which shows a collation object area | region. It is an image figure which shows a mode that the three-dimensional estimated shape and the three-dimensional restoration result were piled up.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2A, 2B ... Camera, 3 ... Image processing part (Attitude estimation means, selection means, 3D estimation means, detection means, determination means).

Claims (6)

Posture estimation means for estimating the posture of the object based on the contour of the object extracted from the captured image of the object;
3D estimation means for estimating a three-dimensional shape of the object based on a posture estimation result by the posture estimation means;
Detecting means for detecting a three-dimensional shape of the object;
A determination unit that determines the validity of the posture estimation result by the posture estimation unit by collating the three-dimensional shape estimated by the 3D estimation unit with the detection result by the detection unit;
An image processing apparatus comprising:   The determination unit determines the validity of the posture estimation result by comparing the three-dimensional shape estimated by the 3D estimation unit and the detection result with respect to a preset comparison target region in the object. The image processing apparatus according to claim 1.   3. The image processing apparatus according to claim 2, further comprising selection means for selecting, as the comparison target area, an area having a degree of overlap lower than a predetermined value in an image obtained by superimposing a plurality of images obtained by imaging the object from a plurality of viewpoints. An image processing apparatus according to 1. A posture estimation step of estimating the posture of the object based on the contour of the object extracted from the captured image of the object;
A 3D estimation step for estimating a three-dimensional shape of the object based on a posture estimation result in the posture estimation step;
A detecting step for detecting a three-dimensional shape of the object;
A determination step of determining the validity of the posture estimation result in the posture estimation step by collating the three-dimensional shape estimated in the 3D estimation step with the detection result in the detection step;
An image processing program for causing a computer to execute.   In the determination step, the validity of the posture estimation result is determined by collating the three-dimensional shape estimated in the 3D estimation step with the detection result for a predetermined collation target region in the object. The image processing program according to claim 4.   A computer is further configured to perform a selection step of selecting an area whose overlapping degree is lower than a predetermined value in an image obtained by superimposing a plurality of images obtained by imaging the object from a plurality of viewpoints as the comparison target area. The image processing program according to claim 5.

Images