JP2015102928A - Processing apparatus, robot, position and attitude detection method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a processing apparatus, robot, position and attitude detection method, program, and the like accurately detecting the position and attitude in six degrees of freedom of an object by using a phase restriction correlation method and a technique involving update processing of a position and attitude based on similarity (in a narrow sense, an asymptotic method).SOLUTION: A processing apparatus 100 includes: an input image acquisition unit 110 that acquires an input image; a template image acquisition unit 120 that sets attitude information on a three-dimensional model data of an object to acquire a template image; and a processing unit 130 that performs phase restriction correlation processing to the input image and template image to determine the similarity between the images, performs update processing of attitude information on the basis of the similarity, and performs phase restriction correlation processing to a new template image and input image acquired according to the attitude information after the update processing, and thereby performing detection processing of the position and attitude of the object.

Description

  The present invention relates to a processing device, a robot, a position / orientation detection method, a program, and the like.

  A position / orientation detection method for detecting a given object at which position in space and in which attitude is useful in various situations. For example, when performing some work using a robot, the position and orientation of the work object are detected in order to perform an appropriate work on the work object. In normal robot work, the position and orientation of the work object must be accurately grasped, such as screwing the work object to the desired position or holding the desired position with the desired hand posture. This is because there are many desirable situations.

  On the other hand, a phase-only correlation method is known as a technique for performing image matching with high robustness and high accuracy (for example, in units of subpixels). For example, Non-Patent Document 1 discloses a detailed method for the phase limiting method. Patent Document 1 discloses a technique for performing estimation processing in units of subpixels, although not limited to the phase-only correlation method.

JP 2009-282635 A

Takafumi Aoki, Koichi Ito, Takuma Shibahara, Kiyoshi Nagashima, "High-Precision Machine Vision Based on Phase-Only Correlation-Towards Image Sensing Technology that Crosses Pixel Resolution-," IEICE Fundamentals Review, Vol. 1, No. 1, pp. 30--40, July 2007.

  Conventionally, when estimating the three-dimensional position and orientation of an object, the three-dimensional shape of the object is measured using a laser range sensor, projection of pattern light by a projector, or a distance image sensor by a time of flight method. The position and orientation of the object are detected by collating with the three-dimensional model data (model data of a three-dimensional shape such as CAD data). That is, the conventional method uses dedicated hardware (a dedicated sensor or a projector that projects pattern light), and uses an input image (for example, a captured image obtained by capturing an object) that is simple two-dimensional image data. Things weren't the main thing.

  The phase-only correlation method is a technique for performing image matching with high accuracy, but using the phase-only correlation method, the position and orientation of an object (in a narrow sense, the position on three axes and the rotation around each axis) A method for estimating information on 6 degrees of freedom is not disclosed. Furthermore, there is no disclosure of a technique that uses the asymptotic method (nonlinear programming, hill-climbing method) for searching for a solution (extreme value) while updating parameters and the phase-only correlation method.

  According to one aspect of the present invention, an input image acquisition unit that acquires an input image and a third of the positions in a three-dimensional space defined by the first to third axes with respect to the three-dimensional model data of the object 3D model data by setting posture information representing the position of the first axis and the third rotation angle, which are rotation angles around the first to third axes. A template image obtaining unit for obtaining a template image from the image, and performing phase-only correlation processing on the input image and the template image to obtain a similarity between the images, and updating the posture information based on the similarity The position and orientation of the object in the input image by performing the phase-only correlation process on the new template image acquired from the orientation information after the update process and the input image Related to processing apparatus including a processing unit that performs detection processing, the.

  In one aspect of the present invention, the position and orientation of an object are detected by performing a phase-only correlation process on a template image and an input image obtained by setting the position and orientation of the three-dimensional model data to obtain a similarity. To do. At this time, the position / orientation update process is performed using the similarity, and the phase limiting process is further performed using the template image acquired from the updated position / orientation. This makes it possible to accurately detect the position and orientation of an object with 6 degrees of freedom by using a phase-only correlation method and a method (asymptotic method in a narrow sense) that involves a position and orientation update process based on similarity. Become.

  Further, in one aspect of the present invention, when the posture information that maximizes the degree of similarity is obtained by the phase-only correlation processing that is performed a plurality of times by the update processing of the posture information, The template image acquired based on the posture information, the position on the first axis and the position on the second axis, which are obtained by the phase-only correlation process on the input image, and the posture information. Information represented by the position on the third axis and the first to third rotation angles represented by the posture information may be detected as the position and posture of the object in the input image. .

  Accordingly, it is possible to appropriately determine each of the six degrees of freedom of the position and orientation based on the position and orientation that maximizes the similarity.

  In the aspect of the invention, the template image acquisition unit is acquired by changing a position on the third axis among the first template image obtained from the posture information and the posture information. The second template image and the third template image acquired by changing the first rotation angle among the posture information and the second rotation angle among the posture information are changed. A fourth template image obtained by the above and a fifth template image obtained by changing the third rotation angle among the posture information, and the processing unit is obtained. The update processing of the posture information based on the first to fifth template images and the plurality of similarities obtained by the phase-only correlation processing using the input image It may be carried out.

  This makes it possible to perform position and orientation update processing using a plurality of template images acquired by changing four degrees of freedom of the position and orientation, respectively.

  In the aspect of the invention, the template image acquisition unit acquires the template image by performing a perspective transformation process of the three-dimensional model data, and the third axis is in a depth direction in the perspective transformation. It may be a corresponding axis.

  Thereby, it is possible to obtain a template image by the perspective transformation process of the three-dimensional model data with the third axis as the axis in the depth direction.

  In the aspect of the invention, the processing unit may perform a phase-only combining process between an input phase image obtained by the frequency conversion process on the input image and a template phase image obtained by the frequency conversion process on the template image. After performing a given weighting process on the result, the similarity may be obtained by performing an inverse conversion process of the frequency conversion process.

  This makes it possible to perform weighting processing in the phase only correlation processing.

  In one aspect of the present invention, the processing unit performs a resolution reduction process for reducing the resolution of the input image to obtain a low-resolution input image having a lower resolution than the input image. The phase only correlation process may be performed on the low resolution input image.

  As a result, it is possible to use a low-resolution input image as the target of the phase-only correlation process.

  In the aspect of the invention, the processing unit performs the resolution reduction process on the template image to obtain a low-resolution template image having the resolution corresponding to the low-resolution input image. The phase only correlation process may be performed on the resolution input image and the low resolution template image.

  Thereby, it becomes possible to make the resolution of a low resolution input image correspond to the resolution of a low resolution template image.

  In one aspect of the present invention, the template image acquisition unit acquires, as the template image, an image in which the resolution of the object in the template image corresponds to the resolution of the object in the low-resolution input image, The processing unit may perform the phase only correlation process on the low resolution input image and the template image.

  Thereby, it becomes possible to make the resolution of a low resolution input image correspond to the resolution of a template image.

  In the aspect of the invention, the processing unit may execute the phase-only correlation process on the input image and the template image after executing the phase-only correlation process on the low-resolution input image.

  This makes it possible to change whether or not to execute the resolution reduction process according to the situation.

  Another aspect of the present invention is an input image acquisition unit that acquires an input image and a position in a three-dimensional space defined by first to third axes with respect to the three-dimensional model data of an object. By setting posture information representing the position on the third axis and the first to third rotation angles which are rotation angles around the respective axes of the first to third axes, A template image acquisition unit for acquiring a template image from the dimensional model data; and performing phase-only correlation processing on the input image and the template image to obtain a similarity between the images, and the posture information based on the similarity And performing the phase-only correlation process on the new template image obtained from the posture information after the update process and the input image, so that the object in the input image A processing unit that performs detection processing of postures relates to a robot comprising a.

  In one aspect of the present invention, the position and orientation of an object are detected by performing a phase-only correlation process on a template image and an input image obtained by setting the position and orientation of the three-dimensional model data to obtain a similarity. To do. At this time, the position / orientation update process is performed using the similarity, and the phase limiting process is further performed using the template image acquired from the updated position / orientation. This makes it possible to accurately detect the position and orientation of an object with 6 degrees of freedom by using a phase-only correlation method and a method (asymptotic method in a narrow sense) that involves a position and orientation update process based on similarity. Become.

  In another aspect of the present invention, input image acquisition processing for acquiring an input image is performed, and the three-dimensional model data of the object is defined in a three-dimensional space defined by the first to third axes. By setting posture information representing the position of the third axis among the positions and the first to third rotation angles that are rotation angles around the respective axes of the first to third axes. Performing a template image acquisition process for acquiring a template image from the three-dimensional model data; performing a phase-only correlation process on the input image and the template image to obtain a similarity between the images; And performing the phase-only correlation process on the new template image acquired from the posture information after the update process and the input image. On the image Kicking relating to the position and orientation detecting method comprising the performing the detection processing of the position and orientation of the object.

  In one aspect of the present invention, the position and orientation of an object are detected by performing a phase-only correlation process on a template image and an input image obtained by setting the position and orientation of the three-dimensional model data to obtain a similarity. To do. At this time, the position / orientation update process is performed using the similarity, and the phase limiting process is further performed using the template image acquired from the updated position / orientation. This makes it possible to accurately detect the position and orientation of an object with 6 degrees of freedom by using a phase-only correlation method and a method (asymptotic method in a narrow sense) that involves a position and orientation update process based on similarity. Become.

  Another aspect of the present invention is an input image acquisition unit that acquires an input image and a position in a three-dimensional space defined by first to third axes with respect to the three-dimensional model data of an object. By setting posture information representing the position on the third axis and the first to third rotation angles which are rotation angles around the respective axes of the first to third axes, A template image acquisition unit for acquiring a template image from the dimensional model data; and performing phase-only correlation processing on the input image and the template image to obtain a similarity between the images, and the posture information based on the similarity And performing the phase-only correlation process on the new template image obtained from the posture information after the update process and the input image, so that the object in the input image As a processing unit that performs detection processing of the postures, there is provided a program causing a computer to function.

  In one aspect of the present invention, the position and orientation of an object are detected by performing a phase-only correlation process on a template image and an input image obtained by setting the position and orientation of the three-dimensional model data to obtain a similarity. Make the computer execute the technique. At this time, the position / orientation update process is performed using the similarity, and the phase limiting process is further performed using the template image acquired from the updated position / orientation. This makes it possible to accurately detect the position and orientation of an object with 6 degrees of freedom by using a phase-only correlation method and a method (asymptotic method in a narrow sense) that involves a position and orientation update process based on similarity. Become.

  As described above, according to some aspects of the present invention, using a phase-only correlation method and a method (positioning and asymptotic method in a narrow sense) that involves position and orientation update processing based on similarity, It is possible to provide a processing device, a robot, a position and orientation detection method, a program, and the like that detect the position and orientation with high accuracy.

FIGS. 1A and 1B are explanatory diagrams of changes in position and orientation of three-dimensional model data and changes in objects in a template image. A three-dimensional graph showing the relationship between posture information and similarity. The figure explaining the sensitivity of the similarity with respect to the attitude | position change at the time of using a phase only correlation method. An example in which the peak position is obtained by interpolation processing. Explanatory drawing of the asymptotic method (mountain climbing method). The structural example of the processing apparatus which concerns on this embodiment. A comparison example of a position where the asymptotic method can be executed and a position where the asymptotic method cannot be executed. The figure explaining the process of the normal phase only correlation method. The example which performs a weighting process with respect to the result of a phase only synthetic | combination process. FIG. 10A shows an example of similarity to posture information when weighting processing is not performed, and FIG. 10B shows an example of similarity to posture information when weighting processing is performed. FIGS. 11A to 11C show examples of similarity to posture information when parameters in the weighting process are changed. 12A to 12C show examples of an input image and a low resolution input image. FIG. 13A to FIG. 13C are diagrams for explaining a difference in similarity to posture information between when the resolution reduction process is not performed and when it is performed. The example of the convergence of attitude | position information at the time of performing the method of this embodiment. The other structural example of the processing apparatus which concerns on this embodiment. 2 is a configuration example of a robot according to the present embodiment. The example of the structure of the robot which concerns on this embodiment. The other example of the structure of the robot which concerns on this embodiment. The example which comprises the processing apparatus etc. which concern on this embodiment with a server system. 20A to 20C are explanatory diagrams of the robustness of the phase only correlation method. The figure explaining the processing content of a phase only correlation method. The flowchart explaining the process which concerns on this embodiment. The figure explaining the production | generation method of a template image, and the change direction in each freedom degree. 24A to 24C are explanatory diagrams of the first and second template images. FIG. 25A to FIG. 25F are explanatory diagrams of third to fifth template images.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, the method of this embodiment will be described. As described above, detection of the three-dimensional position and orientation of an object is required in various situations. On the other hand, there is a known method for acquiring 3D shape information of an object by using a dedicated sensor such as a laser range sensor or using a time of flight method that measures the time from emission to reception of irradiation light. It has been. In these methods, in order to acquire three-dimensional shape information, it is necessary to use a dedicated sensor, equipment (for example, an infrared light irradiation device used in the time of flight method), or the like.

  There is also known a method of measuring a three-dimensional shape of an object by irradiating a target with light of a specific pattern and performing image processing on the pattern on the image captured in the captured image. In this case, a three-dimensional shape is measured from a two-dimensional captured image, but a projector that generates and emits specific pattern light is required.

  Therefore, the present applicant does not use dedicated hardware for three-dimensional position and orientation, and is an input image (captured image in a narrow sense) including an object and ideal three-dimensional shape information of the object. A method for detecting the three-dimensional position and orientation of an object based on the three-dimensional model data is proposed. Specifically, the position and orientation of the object are detected by generating a two-dimensional template image from the three-dimensional model data and performing matching processing between the input image and the template image.

  Various methods for acquiring (generating) a template image from the three-dimensional model data are conceivable. For example, in a three-dimensional space defined by the x-axis, the y-axis, and the z-axis as shown in FIG. A technique may be used in which an image that is arranged at a given position on the z-axis and images the origin direction is used as a template image. At this time, if the upper direction of the template image is the positive y-axis direction, the image captured by the virtual camera is an image as shown in FIG. Specifically, the imaging by the virtual camera is realized by a perspective transformation process or the like.

  In this case, if the position on the x-axis of the three-dimensional model data is changed, the object in the template image moves in the horizontal direction of the image. Specifically, if the position of the object is changed in the direction of the arrow in FIG. 1A, the object in the template image also moves in the direction of the arrow. Similarly, if the position on the y-axis is changed, the object moves in the vertical direction of the image. Further, when the object is moved in the z-axis direction, the distance between the object and the virtual camera changes, so that the size of the object in the template image changes. Also, if the rotation angle u about the x axis, the rotation angle v about the y axis, and the rotation angle w about the z axis are changed, the posture of the object with respect to the virtual camera changes, and thus the object has rotational symmetry, etc. Except for cases, the shape of the object in the template image basically changes. In FIGS. 1A and 1B, the virtual camera is fixed to the coordinate system and the 3D model data side is moved. However, the virtual camera may be moved while the object is fixed.

  That is, when detecting the position and orientation of the object using the template image acquired from the three-dimensional model data and the input image, the position and orientation (x, y, z, u, v, w) of the three-dimensional model data are used. By changing, a plurality of template images having different object positions, sizes, and shapes may be acquired, and an image closest to the input image may be searched among the plurality of template images. In a situation where the template image and the input image are close (in the narrow sense, match), the relative position and orientation of the three-dimensional model data with respect to the virtual camera, and the relative position and orientation of the imaging unit that captured the input image and the actual object Can be considered to be close enough (in the narrow sense to match).

  In general, since image matching requires a similarity that is a parameter indicating how close two images are, the position and orientation are detected by the position and orientation (x, y, z) of the three-dimensional model data that maximizes the similarity. , U, v, w). If (x, y, z, u, v, w) is obtained, the actual object for the imaging unit that captured the input image using the relative position and orientation relationship of the three-dimensional model data with respect to the virtual camera at that time is obtained. Can be obtained. Further, if the arrangement position and orientation of the imaging unit in a given coordinate system are known, it is easy to convert the position and orientation of the object into information of the given coordinate system.

  However, the above is a description of a general method, and the method proposed by the present applicant is characterized in that the above-described phase-only correlation method is used as an image matching method for matching a template image and an input image.

  The phase only correlation method is an image matching method having high robustness and high accuracy as disclosed in Non-Patent Document 1. As will be described later with reference to FIG. 21 and the like, the phase-only correlation method can detect a positional shift on the image plane between the input image and the template image. In other words, regarding the position on two axes out of the three axes positions, for example, the position of the x axis and the y axis, a plurality of template images are acquired by setting various values, and the closest to the input image (x, The process of searching for y) is not necessary. If a predetermined value such as (x, y) = (0, 0) is used when the template image is acquired, the difference between the predetermined value and the position / orientation of the object is a natural result of the phase-only correlation process. It is possible to ask for. That is, in the method of the present embodiment, the detection of the position and orientation of the object is not a problem of obtaining the position and orientation (x, y, z, u, v, w) of the three-dimensional model data that maximizes the similarity, but is similar. It can be handled as a problem of finding (z, u, v, w) that maximizes the degree.

  Specifically, one template image is acquired by determining one set of values of (z, u, v, w). At this time, a predetermined value may be used for (x, y). Then, by performing the phase-only correlation method using the template image and the input image, the relationship between (x, y) and r as shown in FIG. 8 is obtained, so that the maximum value of r is determined as the (z, u, Let the similarity r correspond to v, w). Thus, one combination of the values of (z, u, v, w) and r can be obtained by one phase-only correlation process. This corresponds to, for example, obtaining one bar among a plurality of bars (cuboids) shown in the bar graph of FIG. However, FIG. 2 shows the relationship between (u, v) and r as a three-dimensional graph for the sake of convenience, and processing is performed in a five-dimensional space using (z, u, v, w) and r. . In this specification, the description will be given also using FIG. 10 (A) and the like using a three-dimensional graph by (u, v) and r, but it is extended to a five-dimensional space by (z, u, v, w) and r. Thus, the process is the same.

  This process is performed for each of a plurality of different (z, u, v, w), and a plurality of combinations of parameter (z, u, v, w) and similarity r are obtained, as shown in FIG. A graph with parameters and r can be generated. After that, the parameter that maximizes the similarity may be determined from here, and (u, v) = (0, 0) in the example of FIG. Further, among the positions and orientations having six degrees of freedom, for (x, y), the value obtained by the phase-only correlation process corresponding to the parameter with the maximum similarity may be used.

  Here, it is known that the phase-only correlation method is very sensitive to changes in position and orientation. For example, as will be described later with reference to FIG. 8, the similarity value on the xy plane is a very large value at a position corresponding to the actual shift amount, whereas x and y are slightly shifted. Only the value of the similarity is very small. This also applies to the shape and orientation of the object in the image. For example, in the case where the object in the input image is rotated with respect to the template image, the similarity obtained by the phase-only correlation method is obtained. The value will be very small.

  Therefore, when the size or shape of the object in the template image changes due to the change in the value of the posture information (z, u, v, w), the similarity value for the change is changed as shown in FIG. The change is very steep. Note that the posture information is originally four-dimensional, but in FIG. 3, it is expressed in one dimension for the sake of simplicity. Therefore, when a template image is generated statically, for example, by changing values at predetermined intervals for each of z, u, v, and w, various combinations of (z, u, v, w) In the case of generating a template image, a huge amount of template images are required.

For example, in the case of the rotation angle uvw, it has been confirmed that the similarity value is so small that it cannot be discriminated as an error only by deviating about ± 1 degree from the actual object angle. For example, when the similarity value that can be discriminated as an error in FIG. 3 is r _min and the parameters corresponding to r _min (rotation angles here) are p1 and p2, the interval between p1 and p2 is about 2 degrees (p _max It is only ± 1 degree with respect to. Therefore, with respect to uvw, a template image in which a significant similarity value that can be distinguished from an error is obtained unless the interval is set to at least two degrees or less to generate a template image, that is, the parameter value is There may be no template image larger than p1 and smaller than p2.

Further, since the parameter to be originally obtained is p _max , even if there is at least one template image between p1 and p2, the template image corresponds to p _max , that is, a template image that maximizes the degree of similarity. There is no guarantee. Therefore, if it is desired to obtain a parameter with high accuracy and maximum similarity, the parameter change interval must be very narrow so that a sufficient number of template images are included between p1 and p2. For example, if at least 10 template images are to be included between p1 and p2, it is necessary to generate template images at intervals of about 2/10 degrees for each of u, v, and w. The change width of z is the same in that it must be very narrow although the unit is different. As a result, the number of template images that need to be generated and stored becomes enormous.

  It is also conceivable to obtain extreme values by interpolation processing. For example, when the parameters are pi, pj, and pk as shown in FIG. 4, the similarity between the template image obtained from the parameters and the input image is ri, rj, and rk, respectively. In this case, the parameter value pmax at which the similarity reaches a peak is estimated by performing interpolation processing using a quadratic curve, for example, using the three points (pi, ri), (pj, rj), and (pk, rk). can do. In this case, since the parameter value (pi or the like) corresponding to the template image does not necessarily need to be very close to pmax, the parameter change range can be widened, and the number of template images can be reduced. For example, when three points exist between p1 and p2, for uvw, the interval is set to at least 2/3 degrees or less to generate a template image. However, in this case as well, the number of template images can be reduced as compared with the above example, but the number of template images that should be generated and retained is still not a realistic number.

On the other hand, asymptotic methods (nonlinear programming, hill-climbing methods) are widely known as methods for determining a parameter that maximizes a given index value (here, similarity). For example, as shown in FIG. 5, when the horizontal axis is a parameter and the vertical axis is an index value (similarity in this embodiment), the parameter value is maximized (maximum) while changing the value on the horizontal axis. Search for the value p _max of. In this case, first, we obtain the index value _{q n} of a given parameter _{p n,} then obtains the index value _{q n + 1} to change the parameters _{p n + 1.} If q _{n + 1} > q _n, it can be determined that the parameter is changing in a direction approaching the solution, and if q _{n + 1} <q _n, it can be determined that the parameter is changing in a direction away from the solution. That is, in the asymptotic method, the parameter change in the direction approaching the solution is obtained from the calculation result of the index value, the parameter value is brought close to the correct value, and the process ends when the index value reaches the extreme value. In this way, a template image corresponding to the parameters in the vicinity thereof may be dynamically generated based on the starting point of the asymptotic method. Therefore, as described above, it is not necessary to generate and hold a huge number of template images, the processing load for image generation can be reduced, and the capacity of the storage unit or the like is not reduced.

  Therefore, in this embodiment, the position and orientation of the object are detected using the phase only correlation method and the asymptotic method. Specifically, as illustrated in FIG. 6, the processing apparatus 100 according to the present embodiment acquires a template image based on the input image acquisition unit 110 that acquires an input image and the three-dimensional model data of the object. A template image acquisition unit 120 and a processing unit 130 that performs processing for detecting the position and orientation of an object in the input image based on the input image and the template image are included. And the template image acquisition part 120 is the position in the 3rd axis | shaft among the positions in the 3D space prescribed | regulated by the 1st-3rd axis | shaft with respect to 3D model data, 1st-1st The template image is acquired from the three-dimensional model data by setting posture information representing the first to third rotation angles that are rotation angles around the three axes. The processing unit 130 performs phase-only correlation processing on the input image and the template image to obtain a similarity between the images, performs posture information update processing based on the similarity, and performs post-update processing posture. Phase-only correlation processing is performed on the new template image acquired from the information and the input image.

  Furthermore, when the posture information that maximizes the degree of similarity is obtained by a plurality of phase-only correlation processes by the posture information update processing, the processing unit 130 performs processing on the template image and the input image acquired based on the posture information. The first to third positions represented by the position on the first axis and the position on the second axis, the position on the third axis represented by the attitude information, and the position information obtained by the phase only correlation process. The rotation angle is detected as the position and orientation of the object in the input image.

  The position on the first axis here is, for example, the coordinate value x on the x-axis, and the position on the second axis is, for example, the coordinate value y on the y-axis. x and y are determined by the phase only correlation method as described above. The position on the third axis is a coordinate value on the z-axis, for example, and the first to third rotation angles are rotation angles uvw around the xyz axes. (Z, u, v, w) is determined by the asymptotic method as described above.

  In this way, the phase-only correlation method can be used to detect the position and orientation of the object (particularly, the position and orientation with six degrees of freedom). As disclosed in Non-Patent Document 1, since the phase-only correlation method has high robustness and high accuracy, the method of the present embodiment can also have high robustness and high accuracy. Specifically, even if an object other than the desired object is captured in the input image, the position and orientation can be detected, and the accuracy of the detected position and orientation is increased (for example, an angle error of 1 is set). It is possible to make an order of less than degrees).

  However, when the phase-only correlation method and the asymptotic method are used together, the range of parameters (posture information) in which the asymptotic method can be used becomes a problem. As described above with reference to FIG. 5, in the asymptotic method, the change of the parameter value is directed to the direction corresponding to the extreme value by looking at the change of the index value. In other words, if the hill-climbing method is not reached, that is, if the change direction of the index value is not sufficiently clear, the parameter cannot be updated in the extreme direction. For example, when searching for the maximum value of the similarity r by changing the two parameters uv as shown in FIG. 7, when the parameter is changed from the position shown in A1, the index value is changed. Since it is possible to determine whether the direction is to increase or decrease (determine the gradient direction of the mountain), the asymptotic method can be executed. On the other hand, at the position shown in A2, the starting parameter is too far from the correct value to be obtained, so even if the parameter is changed, it is clear what direction the change will bring to the index value. The asymptotic method cannot be executed.

  Considering the above, there is a problem in the phase-only correlation method that the change of the similarity with respect to the change of the parameter is abrupt as shown in FIG. This is because, even if the parameter is slightly deviated from the correct value, the extremum cannot be searched by the asymptotic method as in the example shown in A2 of FIG. For example, if the limit point (the peak of the mountain) at which the asymptotic method can be started is a point corresponding to p1 and p2 in FIG. 3, the parameter at the start of the asymptotic method is a value between p1 and p2. This range is very narrow as described above, and a value of about ± 1 degree is obtained at a rotation angle.

  In other words, in order to properly execute the asymptotic method, for example, as in the above-described example, it is necessary to generate and hold a huge amount of template images so that the parameter change width is within 2 degrees. become. In this case, compared with the case where the asymptotic method is not used, the parameter that maximizes the similarity, that is, the position and orientation of the object can be detected with high accuracy, or the template image required when the same level of accuracy is obtained. Although it is possible to reduce the number, the problem that the number of template images increases must be taken into consideration.

  Further, as a pre-stage of the object position / orientation detection process according to the present embodiment, a method of performing a relatively low-accuracy position / orientation detection process may be considered. In this case, the processing of this embodiment may be performed using the result of the position and orientation detection in the previous stage. For example, the position and orientation detected in the previous stage is used as the start parameter in the asymptotic method. In this way, it is not necessary to store a large number of template images. However, in this case, the position and orientation of the object, which is the processing result in the previous stage, is required to have an accuracy that the error is within ± 1 degree. In other words, since the requirements for the processing in the previous stage are severe, it cannot be said that the situation is preferable.

  Therefore, the present applicant proposes a method of widening the base of the mountain in the asymptotic method (hill climbing method), that is, a method of widening the range of parameters (posture information) that can appropriately execute the asymptotic method.

  Specifically, the processing unit 130 gives a given value to the result of the phase-only synthesis process of the input phase image obtained by the frequency conversion process for the input image and the template phase image obtained by the frequency conversion process for the template image. After the weighting process is performed, the similarity is obtained by performing an inverse conversion process of the frequency conversion process.

  Various contents of the weighting process here are conceivable. For example, a Gaussian filter may be used. In FIG. 8, one template image is created by determining the values of (z, u, v, w), and a normal phase-only correlation method is performed using the template image and the input image (gaussian is applied). No) is an example. As can be seen from FIG. 8, a very sharp peak appears at a given point on the xy plane (similarity value becomes very large), and if the xy value deviates even a little, the similarity value becomes extremely small. Become. By performing processing for each of a plurality of template images obtained from a plurality of sets of (z, u, v, w) using the method of FIG. 8, similarity and parameters (z, u, FIG. 10 (A) shows the relationship with v, w). In this case, the value of r becomes very high at the correct answer (u, v), and decreases rapidly when the value of (u, v) leaves the correct answer. That is, as described above with reference to FIG. 3 and the like, the range in which the asymptotic method can be used is a steep mountain that is very narrow.

  In contrast, FIG. 9 shows an example in which a Gaussian filter is applied after the phase only synthesis process. Compared with FIG. 8, the peak in the three-dimensional space of (x, y, r) is reduced, and as a result, the value of the similarity r corresponding to (u, v) at this time is small. By performing processing for each of a plurality of template images obtained from a plurality of sets of (z, u, v, w) using the method of FIG. 9, similarity and parameters (z, u, FIG. 10B shows the relationship obtained with respect to v, w). As is clear from FIG. 10B, the slope of the mountain becomes gentler than that in FIG. 10A by applying the Gaussian filter. Therefore, the asymptotic method can be executed even from a position far from the peak of the mountain as compared with FIG. 10A, that is, from a parameter having a relatively large difference from the correct answer.

  Various weighting methods can be considered. For example, the parameter σ that determines the characteristics of the Gaussian filter may be changed. FIG. 11A shows an example in which a Gaussian filter is not applied, whereas FIG. 11B shows an example in which a Gaussian filter with σ = 1 is applied, and with a Gaussian filter with σ = 2 applied. Is FIG. 11C. As can be seen from FIGS. 11 (A) to 11 (C), the peak becomes dull as σ increases, and the skirt portion of the mountain tends to spread. However, as can be seen by comparing the values of r in FIGS. 11 (A) to 11 (C), since the peak value of the similarity itself is reduced by increasing σ, various errors such as uniform errors can be obtained. It may start to be difficult to distinguish from errors. In other words, in the weighting process, it is not necessary to dull the peak strongly, and the intensity of the weighting process (here, an index of how much the peak is dulled before the process) needs to be set appropriately. . Alternatively, a filter other than a Gaussian filter may be used as the weighting process, or the weighting process may be realized by a process other than the filter process.

  Further, the method of expanding the base of the mountain is not limited to the weighting process for the result of the phase only synthesis process. Specifically, the processing unit 130 performs a resolution reduction process for reducing the resolution on the input image, acquires a low-resolution input image having a lower resolution than the input image, and performs an operation on the low-resolution input image. The phase only correlation process may be performed.

  Specific examples are shown in FIGS. 12A to 13C. Here, when the resolution of the input image is 512 × 512 (pixels, pixels) as shown in FIG. 12A, a 256 × 256 low-resolution input image as shown in FIG. As shown in (C), a 128 × 128 low-resolution input image is acquired. Then, as shown in FIG. 12A, the input image is used as it is, and the results obtained by performing the phase-only correlation process for various (z, u, v, w) to obtain the similarity are plotted in FIG. A). Similarly, as shown in FIG. 12B, the result of obtaining the 256 × 256 low-resolution input image and performing the phase-only correlation process is FIG. 13B, and as shown in FIG. FIG. 13C shows the result of obtaining a 128 × 128 low-resolution input image and performing phase-only correlation processing.

  Since FIG. 13A is an original state, the peak is steep as described with reference to FIG. On the other hand, as can be seen from FIG. 13B and FIG. 13C, the peak of similarity can be blunted by using a low-resolution input image. Specifically, the lower the resolution, the greater the blunting of the peak. Also in this case, as in the case of performing the weighting process, the range of parameters for which the asymptotic method can be executed is widened.

FIG. 14 is a diagram showing parameter changes in the asymptotic method when the method of the present embodiment using the weighting process and the low resolution input image is executed. As can be seen from the change in the rotation angle u around the x axis in FIG. 14, even if there is an error of about ± 5 degrees, the solution can be converged by the phase-only correlation method and the asymptotic method. It was confirmed. That is, even when template images are generated and held, the parameter interval may be about ± 5 degrees (10 degrees), so the number of template images can be reduced. Specifically, for each of the four-degree-of-freedom parameters (z, u, v, w), the number of template images can be reduced to about 1/5 compared to the above-described example of ± 1 degree. as a result the number of the whole may be a 1/5 ^4.

  In addition, when position and orientation detection processing with relatively low accuracy is performed as a pre-stage of the processing of the present embodiment, an error of about ± 5 degrees can be allowed in the processing, so that the processing load is reduced or the processing time is shortened. Etc. becomes possible.

  Hereinafter, a system configuration example of the processing apparatus 100 and the like according to this embodiment will be described, and then the phase only correlation method will be briefly described. Furthermore, details of the processing of this embodiment will be described with reference to the flowchart of FIG.

2. System Configuration Example FIG. 15 shows a detailed system configuration example of the processing apparatus 100 according to this embodiment. However, the processing apparatus 100 is not limited to the configuration of FIG. 15, and various modifications such as omitting some of these components or adding other components are possible. Note that various modifications can be made in FIG.

  The processing apparatus 100 includes an input image acquisition unit 110, a template image acquisition unit 120, a processing unit 130, and a storage unit 140. The input image acquisition unit 110, template image acquisition unit 120, and processing unit 130 are the same as those described above with reference to FIG.

  The storage unit 140 serves as a work area for the processing unit 130 and the like, and its function can be realized by a memory such as a RAM, an HDD (hard disk drive), or the like. The storage unit 140 stores, for example, 3D model data of an object that is a target of position and orientation detection processing. The three-dimensional model data is, for example, CAD data. The storage unit 140 is connected to the template image acquisition unit 120, and the template image acquisition unit 120 reads the 3D model data from the storage unit 140. The storage unit 140 may store the template image acquired (generated) by the template image acquisition unit 120.

  FIG. 15 illustrates an example in which the input image acquisition unit 110 acquires a captured image from the imaging unit 150 as an input image. In FIG. 15, the imaging unit 150 is provided outside the processing apparatus 100, but the present invention is not limited to this, and the processing apparatus 100 may include the imaging unit 150.

  Although not shown in FIG. 15, an output unit or the like that outputs the position and orientation of the object detected by the processing unit 130 to another device may be included.

  Also, as shown in FIG. 16, the method of the present embodiment includes an input image acquisition unit 110 that acquires an input image, a template image acquisition unit 120 that acquires a template image based on the three-dimensional model data of the object, The present invention can be applied to a robot 30 including a processing unit 130 that performs processing for detecting the position and orientation of an object in an input image based on the input image and the template image. Then, the template image acquisition unit 120 of the robot 30 performs the first axis position on the third axis among the positions in the three-dimensional space defined by the first to third axes with respect to the three-dimensional model data. By setting posture information representing first to third rotation angles that are rotation angles around each axis of the third axis, a template image is acquired from the three-dimensional model data, and the processing unit 130 The phase-only correlation processing is performed on the input image and the template image to obtain the similarity between the images, and the posture information is updated based on the similarity, and is obtained from the posture information after the update processing. Phase-only correlation processing is performed on the new template image and the input image.

  Here, the processing unit 130 controls the robot mechanism (such as the arm 310) using, for example, the position and orientation of the object detected using the phase-only correlation method and the asymptotic method. For example, the position and orientation of an object that is a target of gripping, processing, assembly, and the like are detected, the end effector 319 is moved from the detected position and orientation to a specified work position (gripping position, etc.), and the end effector 319 moves the object. It is conceivable to perform control or the like for performing work on the position.

  Further, the robot 30 may include a storage unit 140 as in the processing apparatus 100 illustrated in FIG. Further, the robot 30 may include an arm 310, an end effector 319, and a hand eye camera 330 as shown in FIG. Here, the arm 310 and the like may be controlled by the processing unit 130. In FIG. 16, the input image acquisition unit 110 acquires a captured image captured by the hand eye camera 330 as an input image. However, the input image acquisition unit 110 is not limited to this, and is an image from another imaging unit provided in the robot 30. May be used as an input image, or an image from an imaging unit provided outside the robot may be used as an input image.

  Here, the robot 30 may be a robot 30 including a control device 200 and a robot body 300 as shown in FIG. With the configuration of FIG. 17, the control device 200 includes the processing unit 130 of FIG. The robot body 300 includes an arm 310 and an end effector 319. This makes it possible to realize a robot that detects the position and orientation of an object and operates according to control using the position and orientation.

  Note that the configuration example of the robot according to the present embodiment is not limited to FIG. For example, as shown in FIG. 18, the robot may include a robot main body 300 and a base unit portion 350. The robot according to the present embodiment may be a double-arm robot as shown in FIG. 18, and in addition to portions corresponding to the head and the torso, the first arm 310-1 and the second arm 310-2 are provided. Including. In FIG. 18, the first arm 310-1 is composed of joints 311 and 313 and frames 315 and 317 provided between the joints. The same applies to the second arm 310-2. Not. 18 shows an example of a double-arm robot having two arms, the robot of this embodiment may have three or more arms.

  The base unit 350 is provided in the lower part of the robot body 300 and supports the robot body 300. In the example of FIG. 18, the base unit portion 350 is provided with wheels and the like so that the entire robot can move. However, the base unit 350 may be configured to be fixed to a floor surface or the like without having wheels or the like. In the robot system of FIG. 18, the robot main body 300 and the control device 200 are integrally configured by storing the control device 200 in the base unit unit 350.

  Alternatively, the processing unit 130 and the like can be realized by a base built in the robot (more specifically, an IC provided on the base) without providing a specific control device as in the control device 200. May be.

  Further, as shown in FIG. 19, the functions of the processing unit 130 or the like (hereinafter referred to as the processing device 100 or the like) included in the processing device 100 or the robot 30 are performed via a network 400 including at least one of wired and wireless. It may be realized by a server 500 that is communicatively connected to an apparatus (because it is a robot 30 in a narrow sense and will be described as a robot below).

  Or in this embodiment, the server 500 which is a processing apparatus may perform a part of processes, such as a processing apparatus of this invention. In this case, the processing is realized by distributed processing with a processing device or the like provided on the robot side.

  In this case, the server 500 which is a processing device performs processing assigned to the server 500 among the processing in the processing device or the like of the present invention. On the other hand, a processing device or the like provided in the robot performs processing assigned to the processing device or the like of the robot among the processing of the processing device or the like of the present invention.

  For example, the processing apparatus or the like of the present invention performs the first to M-th (M is an integer) processes, the first process is realized by the sub-process 1a and the sub-process 1b, and the second process is the sub-process. Consider a case where each of the first to Mth processes can be divided into a plurality of sub-processes, as realized by 2a and sub-process 2b. In this case, the server 500 which is a processing device or the like performs sub-processing 1a, sub-processing 2a,..., Sub-processing Ma, and the processing device provided on the robot side performs sub-processing 1b, sub-processing 2b,. Distributed processing such as performing processing Mb is conceivable. At this time, the processing apparatus according to the present embodiment, that is, the processing apparatus that executes the first to Mth processes may be a processing apparatus that executes the sub-process 1a to the sub-process Ma, A processing device or the like that executes sub-processing 1b to sub-processing Mb may be used, or a processing device or the like that executes all of sub-processing 1a to sub-processing Ma and sub-processing 1b to sub-processing Mb. Furthermore, the processing apparatus according to the present embodiment is a robot control apparatus that executes at least one sub-process for each of the first to M-th processes.

  As a result, for example, the server 500 having a higher processing capability than the robot-side terminal device (for example, the control device 200 of FIG. 17) can perform processing with a high processing load. Further, the server 500 can collectively control the operations of the robots, and for example, it is easy to cause a plurality of robots to perform a cooperative operation.

  In recent years, the production of a small number of various types of parts has been increasing. And when changing the kind of components to manufacture, it is necessary to change the operation | movement which a robot performs. With the configuration as shown in FIG. 19, the server 500 can change the operations performed by the robot in a batch without re-instructing each robot of the plurality of robots. Furthermore, compared with the case where one processing device or the like is provided for each robot, it is possible to greatly reduce the trouble of performing software update of the processing device or the like.

3. Phase Only Correlation Method Next, the phase only correlation method (POC, Phase Only Correlation) will be briefly described. The phase only correlation method is a method for detecting the parallel movement amount of two images using only phase information out of amplitude information and phase information obtained by two-dimensional Fourier transform of two images. As a characteristic of the phase only correlation method, it is known that it exhibits a sharp characteristic with respect to the amount of movement of the image, and is robust against changes in luminance, background fluctuation, hiding, and the like of the image.

  20A to 20C are explanatory diagrams showing that the phase-only correlation method is robust. In this case, as shown in FIG. 20A, the template image includes only the object and is a solid background image, whereas the input image is as shown in FIG. 20B. The background has a complex color shape. The phase-only correlation method can accurately detect the amount of movement even in such a situation, and can actually detect an object at an appropriate position in the input image as shown in FIG. It is.

  FIG. 21 shows an outline of the phase only correlation method. In FIG. 21, a template image and an input image are used as two images, and the movement amount of the object of the input image with respect to the object of the template image is detected.

In the phase only correlation method, first, two-dimensional Fourier transform is performed on each of two images. In FIG. 21, discrete Fourier transform (DFT) is used, but the present invention is not limited to this. Information on the frequency axis obtained by performing the two-dimensional Fourier transform on the template image is G _ref , and information on the frequency axis obtained by performing the two-dimensional Fourier transform on the input image is G _inp . is there.

For example, if the template image uses a coordinate value x in the image (where x is different from x, which is one of the positions and orientations of the six degrees of freedom of the object), the function f (x) and When expressed, G _ref can be obtained by the following equation (1). Here, N is a constant coefficient for normalizing the transformation, and ω is a variable representing the frequency.

The right side of the above equation (1) can be divided into an amplitude portion and a phase portion, and can be expressed as the following equation (2).

G _ref = A (ω) exp (iθ (ω)) (2)
The above formulas (1) and (2) are formulas obtained by performing one-dimensional Fourier transform on one-dimensional information. Actually, image information is two-dimensional, and Fourier transform is also performed in two dimensions. Therefore, it is necessary to consider the following equation (3). However, since the Fourier transform can be easily extended to two dimensions, the following description will be made in one dimension using the above equations (1) and (2).

Here, consider a case where f (x) is moved by d, for example, a case where an object in the image is moved by d. In this case, the Fourier transform for the image signal after movement is expressed by the following equation (4).

As can be seen by comparing the above equation (4) and the above equation (2), it can be seen that the movement of the position is converted into a phase change in the frequency space, and does not affect the amplitude A (ω). In other words, a change in position in the image is converted into a phase, and a change in brightness is converted into an amplitude. If only the phase is processed, the position is detected without being affected by the brightness of the image. Is possible.

That is, the phase part ∠G _ref obtained by taking A (ω) which is the phase part from the above equation (2) is used for the processing. ∠G _ref can be expressed by the following equation (5).

∠G _ref = exp (iθ (ω)) (5)
Similarly to the template image, ∠G _inp which is a phase portion is obtained for the input image. In the phase-only correlation method, phase-only combining processing is performed using ∠G _ref and ∠G _inp, and inverse Fourier transform is performed on the result. Here, the phase-only combining process corresponds to a process of multiplying one phase information with a complex conjugate of the other phase information.

In this case, if the input image is an image displaced by d with respect to the template image, it can be considered in the same manner as the above equation (4), and therefore ∠G _inp is represented by the following equation (6).

∠G _inp = exp (iθ (ω) −2πiωd) (6)
Therefore, the phase only synthesis process is the multiplication of the complex conjugate of the above equation (5) and the above equation (6), so exp (iθ (ω)) is canceled and becomes exp (−2πiωd). By performing inverse Fourier transform on the result of the phase only synthesis process, the following expression (7) is derived.

Here, C is a constant, and δ represents a δ function. That is, a δ function having a peak at the position d from the origin is obtained by performing the phase-only correlation process on the template image and the input image moved by d with respect to the template image. In FIG. 21 as well, the input image is shifted to the left with respect to the template image, but in the processing result, a peak is obtained at a position shifted to the left with respect to the origin.

On the other hand, when there is no correlation between the template image and the input image, since there is no correlation between ∠G _ref and ∠G _inp , for example, ∠G _inp = exp (iφ (ω)). In this case, the result of the phase-only synthesis process is exp (−iθ (ω) + iφ (ω)), which is a random variable, and the result of performing the inverse Fourier transform is uniform noise.

  If the template image and the input image do not have the same object but show a fairly high correlation, the processing result is in an intermediate state between the δ function and the uniform noise. This is because, in the method of the present embodiment, the value of (z, u, v, w) is different from the solution, and the shape, orientation, and size of the object in the template image are slightly different from the input image. Correspond. That is, although the peak appears, the sharpness is lower than the δ function. In the present embodiment, as described above, based on the difference in peak sharpness (difference in similarity value) due to the deviation in posture, the value of (z, u, v, w) that is the solution is searched by the asymptotic method. .

4). Details of Processing Details of processing according to the present embodiment will be described with reference to a flowchart of FIG. When this process is started, first, a template image of an object is generated from the result of position / orientation detection with relatively low accuracy obtained by pre-processing (S101).

  Specifically, the template image acquisition unit 120 acquires the template image by performing a perspective conversion process on the three-dimensional model data. The third axis may be an axis corresponding to the depth direction in perspective transformation. For example, as shown in FIG. 23, the direction of the line of sight from the set viewpoint (the direction of the optical axis when a virtual camera is assumed) may be the direction along the z axis.

  Then, processing using the phase only correlation method (POC) is performed using the generated template image and the input image (S102). At this time, as described above, it is preferable to use a method of expanding the base of the mountain. Specifically, a resolution reduction process may be performed on the input image to obtain a low resolution input image, and POC may be performed on the low resolution input image.

  At this time, the resolution of the input image and the resolution of the template image are required to correspond (in a narrow sense, match). Therefore, the template image used for POC needs to have the same resolution as the low-resolution input image.

  For example, the processing unit 130 performs a resolution reduction process on the template image to obtain a low resolution template image having a resolution corresponding to the low resolution input image, and performs phase conversion on the low resolution input image and the low resolution template image. Limited correlation processing may be performed. In this case, a template image having a high resolution (for example, the same resolution as the input image before the resolution reduction process) is acquired in the process of S101, and the resolution reduction process is also performed on the template image in the process of S102. become.

  Alternatively, at the time of the processing of S101, the template image acquisition unit 120 may acquire, as a template image, an image whose object resolution in the template image corresponds to the object resolution in the low-resolution input image. In this case, in the process of S102, the processing unit 130 performs the phase only correlation process on the low resolution input image and the template image. That is, since it is sufficient that the resolution of the template image and the low-resolution input image (objects included in the narrow sense) correspond to each other, the resolution may be adjusted at the time of obtaining the template image.

  In S102, weighting processing may be performed on the result of the phase-only synthesis processing as another method of expanding the base of the mountain.

  After S102, processing for obtaining a parameter (position and orientation) change direction in a direction approaching the solution by an asymptotic method is performed. Specifically, each of z, u, v, and w is minutely changed from the position and orientation corresponding to S102, and a template image is generated for each (S103).

  That is, the template image acquisition unit 120 obtains the first template image obtained from the posture information, the second template image obtained by changing the position on the third axis in the posture information, and the posture information. 3rd template image acquired by changing the 1st rotation angle among them, 4th template image acquired by changing the 2nd rotation angle among posture information, and posture information And a fifth template image acquired by changing the third rotation angle.

  Then, using each point plate image of the acquired plurality of template images, POC is performed in the same manner as in S102 to obtain a similarity (S104), and from the result, each degree of freedom of (z, u, v, w) is obtained. On the other hand, it is determined what kind of change the similarity can be increased (S105).

  In S <b> 105, for example, the processing unit 130 performs posture information update processing based on the obtained first to fifth template images and a plurality of similarities obtained by phase-only correlation processing using the input image.

  Here, the first template image is an image corresponding to S102, and the change width is zero. Therefore, if the position and orientation in S102 are in the state shown in FIG. 23, the first template image is as shown in FIG. The second template image is an image obtained by slightly changing the position on the z axis in FIG. Note that the direction of change in the z-axis may be closer to the solution when changed in the positive direction, or may be closer to the solution when changed in the negative direction. Therefore, it is preferable to slightly change the z axis on both the plus and minus sides. Specifically, by making minute changes in the directions of B1 and B2 in FIG. 23, as shown in FIGS. 24B and 24C, two template images are used as the second template image. get.

  Further, if the rotation angle u around the x axis is considered as the first rotation angle, u is also slightly changed on both sides of B3 and B4. Therefore, two images shown in FIGS. 25A and 25B are acquired as the third template image. Similarly, by slightly changing the second rotation angle v on both sides of B5 and B6, two images of FIG. 25C and FIG. 25D are acquired as the fourth template image, By slightly changing the rotation angle w of 3 to both sides of B7 and B8, two images shown in FIGS. 25E and 25F are acquired as the fifth template image.

  Note that in FIG. 24B to FIG. 25F, the change width is set large in order to clarify the change in shape, posture, and size of the object on the image with respect to the change in posture information. The change of (z, u, v, w) and the change of the object on the image are very small.

  As a result, for each of (z, u, v, w), it is possible to obtain a change in similarity when the value is changed to both plus and minus. Therefore, in S105, using the result, for each of (z, u, v, w), how much the value is changed in the plus or minus direction, the direction in which the similarity is maximized is changed. And the position and orientation (z, u, v, w) are updated.

  Then, it is determined whether or not the degree of similarity is maximized (S106), and if it is maximized, the processing is terminated assuming that (z, u, v, w) at that time corresponds to the position and orientation of the object. As described above, x and y may be determined from the result of POC using the template image at (z, u, v, w) (x, y that maximizes similarity).

  On the other hand, in the case of No in S106, the posture information is further updated and POC is executed (S107). Specifically, with the posture information updated according to the result of S105 as a reference, values for (z, u, v, w) are slightly changed with respect to the reference, 5 template images may be acquired and the above process repeated.

  Although omitted in the above description, the process of reducing the resolution of the input image does not have to be performed constantly in the process of the present embodiment. For example, the processing unit 130 may execute a process using the phase only correlation method for the input image after the process using the phase only correlation method for the low resolution input image.

  The reason why the resolution of the input image is lowered is to widen the range in which the asymptotic method can be executed as described above. That is, at the start of the asymptotic method or immediately after the start, there is a high possibility that the posture information at that stage is far from the solution value, so that the usefulness of resolution reduction is high. On the other hand, if the asymptotic method is normally executed and the posture information update processing or the like has been performed a certain number of times, the posture information at that stage is considered to be close to charging / decomposition. For example, if the error with respect to the solution is close to about ± 1 degree, it is possible to converge to the solution by an asymptotic method without reducing the resolution and widening the bottom of the mountain.

  As shown in FIGS. 13A to 13C, the base of the mountain is widened by reducing the resolution, but the peak is blurred accordingly. In other words, the accuracy of position and orientation detection may be reduced by performing the resolution reduction process. Therefore, whether or not to execute the resolution reduction process is switched depending on whether it is a situation where it is important to expand the range in which the asymptotic method can be executed or whether the accuracy of position and orientation detection is important. Good. Specifically, as described above, when the first phase is set for a certain period of time (or the number of updates) from the start and the second phase is set thereafter, resolution reduction processing is performed on the input image in the first phase. In the second phase, the resolution reduction process may be skipped.

  As described above, the template image needs to correspond to the image to be matched with the resolution. Therefore, in the first phase, as described above, a template image having a relatively high resolution may be acquired, and a resolution reduction process may be performed on the template image to acquire a low resolution template image. Or you may acquire the image of the resolution corresponding to a low-resolution input image as a template image. In the second phase, since the resolution reduction process is not performed on the input image, an image having a resolution corresponding to the input image may be acquired as a template image. The resolution is the phase-only correlation process in the first phase. The resolution of the template image used in the above is higher.

  As for weighting processing (filter processing using a Gaussian filter in a narrow sense), the weighting processing may be used for noise reduction, so it is basically assumed that the processing is skipped depending on the situation. Not done. However, various modifications can be made in this respect, and skipping the weighting process according to the situation is not prevented.

  Note that the processing apparatus 100 or the like according to the present embodiment may realize part or most of the processing by a program. In this case, a processor such as a CPU executes the program, whereby the processing device 100 according to the present embodiment is realized. Specifically, a program stored in a non-temporary information storage medium is read, and a processor such as a CPU executes the read program. Here, the information storage medium (computer-readable medium) stores programs, data, and the like, and functions as an optical disk (DVD, CD, etc.), HDD (hard disk drive), or memory (card type). It can be realized by memory, ROM, etc. A processor such as a CPU performs various processes according to the present embodiment based on a program (data) stored in the information storage medium. That is, in the information storage medium, a program for causing a computer (an apparatus including an operation unit, a processing unit, a storage unit, and an output unit) to function as each unit of the present embodiment (a program for causing the computer to execute processing of each unit) Is memorized.

  Further, the processing apparatus and the like of the present embodiment may include a processor and a memory. The processor here may be, for example, a CPU (Central Processing Unit). However, the processor is not limited to the CPU, and various processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor) can be used. The processor may be an ASIC hardware circuit. In addition, the memory stores instructions that can be read by a computer. When the instructions are executed by the processor, each unit of the processing apparatus according to the present embodiment is realized. The memory here may be a semiconductor memory such as SRAM or DRAM, or a register or a hard disk. Further, the instruction here may be an instruction of an instruction set constituting the program, or an instruction for instructing an operation to the hardware circuit of the processor.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described at least once together with a different term having a broader meaning or the same meaning in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. Further, the configuration and operation of the processing apparatus 100 and the like are not limited to those described in the present embodiment, and various modifications can be made.

30 robot, 100 processing device, 110 input image acquisition unit,
120 template image acquisition unit, 130 processing unit, 140 storage unit, 150 imaging unit, 200 control device, 300 robot body, 310 arm, 311, 313 joint,
315, 317 frame, 319 end effector, 330 hand eye camera,
350 base unit, 400 network, 500 server

Claims (12)

An input image acquisition unit for acquiring an input image;
For the three-dimensional model data of the object, the position on the third axis among the positions in the three-dimensional space defined by the first to third axes, and each axis of the first to third axes A template image acquisition unit configured to acquire a template image from the three-dimensional model data by setting posture information representing first to third rotation angles that are rotation angles around;
A phase-only correlation process is performed on the input image and the template image, a similarity between the images is obtained, the attitude information is updated based on the similarity, and the attitude information after the update process is used. A processing unit that performs detection processing of the position and orientation of the object in the input image by performing the phase-only correlation process on the acquired new template image and the input image;
The processing apparatus characterized by including. In claim 1,
The processor is
When the posture information that maximizes the degree of similarity is obtained by a plurality of phase-only correlation processing by the update processing of the posture information,
The template image acquired based on the posture information, and the position on the first axis and the position on the second axis, which are obtained by the phase-only correlation process on the input image;
A position on the third axis represented by the posture information;
Information represented by the first to third rotation angles represented by the posture information,
A processing apparatus that detects the position and orientation of the object in the input image. In claim 1 or 2,
The template image acquisition unit
A first template image obtained from the posture information;
Of the posture information, a second template image acquired by changing the position on the third axis;
Of the posture information, a third template image acquired by changing the first rotation angle;
Among the posture information, a fourth template image acquired by changing the second rotation angle;
A fifth template image obtained by changing the third rotation angle in the posture information;
The processor is
The posture information is updated based on the obtained first to fifth template images and a plurality of the similarities obtained by the phase-only correlation processing using the input image. Processing equipment. In any one of Claims 1 thru | or 3,
The template image acquisition unit
The template image is obtained by performing a perspective transformation process of the three-dimensional model data,
The processing apparatus according to claim 3, wherein the third axis is an axis corresponding to a depth direction in the perspective transformation. In any one of Claims 1 thru | or 4,
The processor is
After performing a given weighting process on the result of the phase-only synthesis process between the input phase image obtained by the frequency conversion process on the input image and the template phase image obtained by the frequency conversion process on the template image The processing apparatus is characterized in that the similarity is obtained by performing an inverse conversion process of the frequency conversion process. In any one of Claims 1 thru | or 5,
The processor is
A resolution reduction process for reducing the resolution is performed on the input image to obtain a low-resolution input image having a lower resolution than the input image, and the phase-only correlation process is performed on the low-resolution input image. The processing apparatus characterized by performing. In claim 6,
The processor is
The resolution reduction process is performed on the template image to obtain a low resolution template image having the resolution corresponding to the low resolution input image, and the low resolution input image and the low resolution template image A processing apparatus that performs phase-only correlation processing. In claim 6,
The template image acquisition unit
The resolution of the object in the template image is acquired as the template image an image corresponding to the resolution of the object in the low-resolution input image,
The processor is
A processing apparatus that performs the phase-only correlation process on the low-resolution input image and the template image. In any of claims 6 to 8,
The processor is
The processing apparatus, wherein the phase-only correlation process is performed on the input image after the phase-only correlation process is performed on the low-resolution input image. An input image acquisition unit for acquiring an input image;
For the three-dimensional model data of the object, the position on the third axis among the positions in the three-dimensional space defined by the first to third axes, and each axis of the first to third axes A template image acquisition unit configured to acquire a template image from the three-dimensional model data by setting posture information representing first to third rotation angles that are rotation angles around;
A phase-only correlation process is performed on the input image and the template image, a similarity between the images is obtained, the attitude information is updated based on the similarity, and the attitude information after the update process is used. A processing unit that performs detection processing of the position and orientation of the object in the input image by performing the phase-only correlation process on the acquired new template image and the input image;
A robot characterized by including: Performing input image acquisition processing for acquiring an input image;
For the three-dimensional model data of the object, the position on the third axis among the positions in the three-dimensional space defined by the first to third axes, and each axis of the first to third axes Performing template image acquisition processing for acquiring a template image from the three-dimensional model data by setting posture information representing first to third rotation angles that are rotation angles around;
A phase-only correlation process is performed on the input image and the template image, a similarity between the images is obtained, the attitude information is updated based on the similarity, and the attitude information after the update process is used. Performing detection processing of the position and orientation of the object in the input image by performing the phase-only correlation process on the acquired new template image and the input image;
A position and orientation detection method comprising: An input image acquisition unit for acquiring an input image;
For the three-dimensional model data of the object, the position on the third axis among the positions in the three-dimensional space defined by the first to third axes, and each axis of the first to third axes A template image acquisition unit configured to acquire a template image from the three-dimensional model data by setting posture information representing first to third rotation angles that are rotation angles around;
A phase-only correlation process is performed on the input image and the template image, a similarity between the images is obtained, the attitude information is updated based on the similarity, and the attitude information after the update process is used. As a processing unit that detects the position and orientation of the object in the input image by performing the phase-only correlation process on the acquired new template image and the input image,
A program characterized by operating a computer.