JP2017156968A - Information processing device, information processing method, program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To associate a model feature with image feature quickly and stably, so that position posture of an object is precisely estimated.SOLUTION: An information device includes image acquiring means for acquiring an image that contains an object, feature detection means which detects an image feature that represents the object from the image, selection means which selects reference distance information from between first reference distance information which stores a distance from the detected image feature in a first direction on the basis of a direction of a model feature constituting a model and second reference distance information that stores a distance from the detected image feature in a second direction being different from the first direction, association means which associates an image feature corresponding to the model feature on the basis of the reference distance information selected by the selection means and a position of the model feature being transformed by coordinate transformation means, and derivation means which derives position posture of the object on the basis of the result associated by the association means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a method for associating a shape model with an image feature for measuring the position and orientation of an object to be measured.

  As a technique for estimating the position and orientation of an object without contact, a technique for estimating the position and orientation of a shape model so that the difference between the shape model of the measured object and the object image in the image of the measured object is reduced It has been known. This technique is called model fitting, and is used when the position and orientation of a part (object to be measured) placed at an unspecified position and orientation is measured by a vision sensor, such as bin picking of a part by a robot.

  In model fitting, each geometric feature constituting the shape model is associated with an image feature included in the observed image, and a difference between the geometric feature of the shape model and the image feature is calculated. Then, the non-linear parameter representing the position and orientation of the measured object is repeatedly estimated so that the difference becomes small. Here, associating processing in model fitting requires a search processing amount to be enormous depending on the amount of geometric feature data to be processed. Therefore, a technique for performing the associating processing quickly and efficiently is required.

  Japanese Patent Application Laid-Open No. 2004-133867 discloses a method for performing a high-speed association between a shape model and an image feature when an edge is used as an image feature. In this method, a signed distance field in which a signed distance value from the target pixel to the edge with the shortest distance is stored in each pixel is generated in advance, and the signed distance field is calculated when searching for correspondence between the shape model and the image feature. By referencing, the nearest neighbor search is performed at high speed.

JP 2010-205095 A

  However, the method of Patent Document 1 is a method of simply associating an edge feature (image feature) that is the nearest neighbor on the image with the position of the model edge point (geometric feature). Does not consider the direction. Therefore, there are problems that the result of association includes many errors and the accuracy of association becomes low.

  The present invention has been made in view of the above problems, and an object of the present invention is to accurately associate a model feature with an image feature at high speed and estimate the position and orientation of an object with high accuracy.

  In order to solve the above problems, an information processing apparatus according to the present invention represents, for example, a model holding unit that holds model information of an object, an image acquisition unit that acquires an image including the object, and the object from the image Feature detection means for detecting an image feature; first reference distance information storing a distance to the detected image feature in a first direction based on a direction of the model feature constituting the model information; Selecting means for selecting any reference distance information out of second reference distance information storing a distance to the detected image feature in a second direction different from the first direction; and An association means for associating an image feature corresponding to the model feature based on the selected reference distance information and the position of the model feature transformed by the coordinate transformation means; Based on the result corresponding the marked section, and a derivation unit that derives the position and orientation of said object.

  According to the present invention, model features and image features can be associated with each other quickly and stably, and the position and orientation of an object can be estimated with high accuracy.

It is a figure which shows the structure of the information processing apparatus in the 1st Embodiment of this invention. It is a figure explaining the component of the three-dimensional shape model in the 1st Embodiment of this invention. It is a figure explaining the distance field with a sign according to direction in a 1st embodiment of the present invention. It is a flowchart which shows the process sequence which the information processing apparatus in the 1st Embodiment of this invention performs. It is a figure explaining the production | generation of the signed distance field according to direction in the 1st Embodiment of this invention. It is a figure which shows the structure of the information processing apparatus in the 2nd Embodiment of this invention. It is a flowchart which shows the process sequence which the information processing apparatus in the 2nd Embodiment of this invention performs. It is a figure which shows the structure of the information processing apparatus in the 3rd Embodiment of this invention. It is a figure which shows the hardware block diagram of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  First, before describing the details of each embodiment of the present invention, the hardware configuration of the information device of each embodiment will be described with reference to FIG. In the figure, a CPU 910 comprehensively controls devices connected via a bus 900. The CPU 910 reads and executes processing steps and programs stored in a read-only memory (ROM) 920. In addition to the operating system (OS), each processing program, device driver, and the like according to the present embodiment are stored in the ROM 920, temporarily stored in a random access memory (RAM) 930, and appropriately executed by the CPU 910. The input I / F 940 inputs an image from an external device (such as an imaging device) as an input signal in a format that can be processed by the information processing device 1. Further, the output I / F 950 outputs, as an output signal, a feature associated with the external device in a format that can be processed by the external device.

[Embodiment 1]
In the present embodiment, the information processing apparatus and method according to the present invention are obtained from a model edge point in a three-dimensional shape model and an input image in model fitting based on a three-dimensional shape model of an object to be measured and an edge feature in the image. A case where the method is applied to a method of associating detected edge features will be described.

  In the present embodiment, a plurality of signed distance fields (hereinafter referred to as direction-specific signed distance fields) are generated by limiting the number of search directions by the number of preset search directions. Then, by selecting and referencing a direction-specific signed distance field that is close to the normal direction on the model edge point image from the direction-specific signed distance field, Refers to neighboring edge features. As a result, it is possible to perform the nearest neighbor search process with high matching accuracy at high speed. An example of using the model fitting to estimate the position and orientation of an object to be measured is given.

  Next, a functional configuration of the information processing apparatus 1 in the present embodiment will be described with reference to FIG. As illustrated in FIG. 1, the information processing apparatus 1 includes an image acquisition unit 110, a shape model holding unit 120, a feature detection unit 130, a direction-specific reference map generation unit 140, a coordinate conversion unit 150, a direction-specific reference map selection unit 160, An association unit 170, a residual calculation unit 180, and a position / orientation derivation unit 190 are provided.

  Each of these functional units is realized by the CPU 910 in FIG. 9 described above expanding the program stored in the ROM 920 in the RAM 930 and executing processing according to each flowchart described later. Further, for example, when hardware is configured as an alternative to software processing using the CPU 910, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

  The information processing apparatus 1 is connected to an external imaging apparatus 10. The imaging device 10 is a camera that captures a normal image. The captured image may be a gray image or a color image. Further, each pixel may be a distance image having depth information. As long as an image of the object to be measured is obtained, there is no limitation on the captured image, and any image may be used. In the present embodiment, the imaging device 10 is a camera that captures a grayscale image in which a luminance value is stored in each pixel, and outputs a grayscale image. Further, the internal parameters of the imaging apparatus 10 according to the present embodiment such as the focal length, the principal point position, and the lens distortion parameter are referred to in advance by the method disclosed in the following document, referring to the specification of the device to be used. Calibrate in advance.

R. Y. Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shel TV encer and en l RA-3, no. 4, 1987.
Next, each part which comprises the information processing apparatus 1 is demonstrated.

  The image acquisition unit 110 acquires an image captured by the imaging device 10. As described above, in the present embodiment, the imaging device 10 is a camera that captures a grayscale image, and the image acquisition unit 110 acquires a grayscale image via an interface.

  The shape model holding unit 120 holds a three-dimensional shape model 20 that represents the shape of the object to be measured. In this embodiment, a three-dimensional shape model (for example, a CAD model or a polygon model) is a local three-dimensional structure composed of a three-dimensional position and a three-dimensional direction in a model coordinate system as shown in FIG. It is assumed to be constituted by line segment information (hereinafter referred to as model features or model edge points). However, the shape information held as the three-dimensional shape model is not particularly limited in the expression format as long as it includes the three-dimensional coordinate information representing the shape or feature of the object to be measured. For example, a simple 3D point set, a 3D line set representing a ridgeline, a 3D surface and a polygonal shape information represented by a set of lines and 3D points, etc. May be. A three-dimensional shape model may be expressed as a set of polygons composed of three points, three sides, and one surface, or a three-dimensional shape may be expressed as a set of simple three-dimensional points. Alternatively, a method of expressing a three-dimensional shape parametrically by a combination of implicit functions may be used. As long as it conforms to the shape of the object to be measured, there is no particular limitation on the method of expressing the three-dimensional shape model. The three-dimensional shape model held in the shape model holding unit 120 is input to the coordinate conversion unit 150.

  The feature detection unit 130 detects image features from the two-dimensional image input from the image acquisition unit 110. In this embodiment, an edge feature is detected as an image feature. The edge feature represents a point where the luminance in the image changes discontinuously, and a position where the luminance gradient in the image becomes an extreme value is defined as an edge position, and a normal line of the gradient direction is defined as an edge direction. However, the image feature of the present embodiment is not limited to the edge feature. For example, a feature point such as a Harris feature point or a FAST feature point may be detected as the image feature. In this case, the 3D shape model of the measured object is represented by 3D point cloud data, and the position and orientation are calculated based on the correspondence between the feature points and the 3D points. Further, instead of using only specific features, a plurality of features (feature points and edges) may be used for calculating the position and orientation. As described above, the image feature is not particularly problematic as long as it is a feature corresponding to the model points constituting the three-dimensional shape model. Details of the feature detection processing will be described later.

  The direction-specific reference map generation unit 140 generates a direction-specific reference map (reference distance information) for searching for edge features corresponding to model edge points at high speed based on the edge features detected by the feature detection unit 130. The direction-specific reference map is a map that stores information for referring to each pixel (each element) constituting the image and the position of an edge feature that is close in a predetermined search direction. Here, the predetermined search direction in the present embodiment represents a direction on an image for searching for edge features. In this embodiment, a direction-specific distance field is generated as a direction-specific reference map. The direction-specific signed distance field is an image in which a signed distance from a pixel to the nearest edge feature is held as a pixel value along a target search direction.

  FIG. 3 shows an example of a signed distance field by direction. 4A shows the lateral direction, FIG. 4B shows the vertical direction, FIG. 4C shows the diagonal upper right direction, and FIG. 4D shows the signed distance field by direction with respect to the diagonal lower right direction. Depending on the normal direction of the model point on the image, refer to the signed distance field by direction of the appropriate search direction for each model point, so that the position of the edge feature closest to the model point in the normal direction can be determined. It becomes possible to search at high speed. In the present embodiment, a direction-specific signed distance field for four directions in the horizontal direction, the vertical direction, the diagonally upper right direction, and the diagonally lower right direction on the image is generated as the direction-specific distance field. However, the direction and the number to be generated as the direction-specific distance field are not limited to the above. For example, the direction-specific distance field including only the vertical and horizontal directions on the image may be generated. The number of diagonally divided distance fields may be generated by increasing the number of divisions in the oblique direction. As long as the direction-specific signed distance field for a plurality of search directions is generated, there is no limitation on the number of generated methods and directions. In addition, an image that holds a signed distance field for each direction may be generated for each direction, or a pixel value of a signed distance field for each direction is held as multi-plane information such as RGB information. The information on the signed distance field for each direction may be held as one image. As long as the information of the direction-specific signed distance field regarding a plurality of directions is held, the data format for holding the direction-specific signed distance field is not limited, and any data format may be used. Details of the process for generating the direction-specific signed distance field will be described later.

The coordinate conversion unit 150 converts the position and direction of the model point represented in the model coordinate system to the position and direction in the image coordinate system based on the approximate values of the position and orientation of the measurement object and the internal parameters of the imaging device 10. Perform conversion. Specific processing of the coordinate conversion processing will be described later.
The direction-specific reference map selection unit 160 selects a direction-specific reference map that is closest to the normal direction of the model point based on the direction of the model point in the image coordinate system obtained by the coordinate conversion unit 150. Details of the selection process of the reference map for each direction will be described later.

  The associating unit 170 refers to the nearest edge feature from the reference map by direction selected by the map selecting unit 160 based on the position of the model point in the image coordinate system obtained by the coordinate converting unit 150. The model points are associated with the edge features. Details of the association processing will be described later.

  The residual calculation unit 180 acquires a model edge point and a pair of edge features associated with the model edge point from the association unit 170 of the information processing apparatus 1, and the distance on the image between the model edge point and the edge feature. (Hereinafter referred to as residual) is calculated. Details of the processing will be described later.

  The position / orientation deriving unit 190 derives the position and orientation between the imaging apparatus 10 and the object to be measured so that the residual is minimized based on the residual information calculated by the residual calculating unit 180 (position Posture estimation). Details of the processing will be described later.

  The above is the description of an example of the configuration of the information processing apparatus 1.

  Next, a processing procedure according to the present embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing a procedure for feature detection in the present embodiment.

(Step S1100)
First, in step S1100, the coordinate conversion unit 150 acquires coordinate conversion parameters for geometrically converting the position and direction of the model point expressed in the model coordinate system to the same image coordinate system as the image feature. Specifically, an approximate value of the position and orientation of the object to be measured with respect to the imaging apparatus 10 is input as a coordinate conversion parameter. In this embodiment, assuming that the approximate position and orientation where an object is placed are known in advance, the value is used as an approximate value. However, the method for setting the approximate value of the position and orientation is not limited to the above method. For example, when measuring the position and orientation of an object continuously in the time axis direction, the position and orientation of the previous (previous time) Measurement values may be used as approximate positions and orientations. Alternatively, the rough position and orientation of the measured object may be estimated by holding images of the measured object captured in various postures as templates and performing template matching on the input image. As long as the position and orientation of 6 degrees of freedom of the measurement object can be obtained, the input coordinate conversion parameters are not limited and may be in any format. When the process of step S1100 is completed, the process proceeds to step S1200.

(Step S1200)
Next, in step S1200, the image acquisition unit 110 acquires a grayscale image from the imaging device 10. Even if the image is not directly acquired from the imaging device 10, a gray image may be acquired by storing an image captured in advance in a storage medium and reading the storage medium. When the acquisition of the grayscale image is completed, the image acquisition unit sends the acquired grayscale image to the feature detection unit. The process of step S1200 is terminated and the process proceeds to step S1300.

(Step S1300)
Next, in step S1300, the feature detection unit 130 detects image features from the grayscale image acquired in step S1200. In the present embodiment, as described above, an edge is detected as an image feature. An edge is a point that is an extreme value of the density gradient. In this embodiment, as an edge detection method, the Canny edge algorithm disclosed in the following document is used.

  Canny, J, “A Computational Approach To Edge Detection”, IEEE Trans. Pattern Analysis and Machine Intelligence, 8 (6): 679-698, 1986.

  As a result of the edge detection process, the position of the pixel serving as the edge in the grayscale image is obtained. Note that the edge detection method is not limited to this, and the edge position may be calculated from the response value using an edge detection filter such as a Sobel filter or a prewitt filter. In addition to the method of detecting edges from the entire image as described above, a method of detecting edges locally in the vicinity of the region of interest as disclosed in the following document may be used.

T. Drummond and R. Cipolla, “Real-time visual tracking of complex structures
", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 7, pp. 932-946, 2002." Transactions on Pattern Analysis and Machine Intelligence, vol.

  As long as the edge feature is detected on the grayscale image and associated with the model edge point, there is no problem even if it is detected by any method, and the selection of the detection method is not particularly limited. If the process of detecting all image edges from the grayscale image is completed, the process of step S1300 is terminated, and the process proceeds to step S1400.

(Step S1400)
Next, in step S1400, the direction-specific reference map generation unit 140 generates a plurality of direction-specific signed distance maps based on the image edges calculated in step S1300. In this embodiment, as described above, a signed distance field for each direction corresponding to a total of four directions in the horizontal direction, the vertical direction, the diagonal upper right direction, and the diagonal lower right direction is generated on the image.

  First, four images having the same size as the image input in step S1200 are generated as images for storing direction-specific distance fields. Then, for all the images, 0 is assigned to the pixels at the positions of all the edge features detected in step S1300, and the other pixels are flagged as unprocessed. Then, for each signed distance field by direction, a signed distance field expansion process that limits the expanding direction is performed. Specifically, for all the unprocessed pixels, adjacent pixels are checked along a limited direction, and when a signed distance value is stored in the adjacent pixel, a signed distance value of the adjacent pixel The signed distance value increased or decreased by 1 is stored in the target pixel. For example, in the case of a signed distance field by direction in the horizontal direction, the left and right adjacent pixels of the target pixel are checked, and the signed distance value of the target pixel is referred to by referring to the signed distance values stored in the left and right adjacent pixels. Is stored. The above process is performed as one expansion process, and the expansion process is repeated for the pixel range to be searched on the image, thereby generating a signed distance field by direction. The number of times of performing the expansion process may be a manually set fixed value, or may be set according to the ratio of the input image size. In addition, when the ambiguity of the approximate position and orientation of the measured object is known, the approximate position and orientation may be set according to the range value projected onto the image. Alternatively, the expansion process may be performed until there is no unprocessed pixel in the directional distance field in each direction. In this embodiment, the number of times of performing the expansion process is set from the number of pixels that is a quarter of the long side of the image size.

  FIG. 5 shows an example of expansion processing of a signed direction distance field in the horizontal direction. First, in the initial state (FIG. 5A), a signed distance value is stored only in the pixel at the edge position. As a result of one expansion process (FIG. 5B), a signed distance value of +1 or −1 corresponding to the direction is stored in a pixel adjacent in the horizontal direction to the pixel at the edge position. Then, a signed distance field expanded in the horizontal direction is generated by repeating the expansion process twice (FIG. 5C) and three times (FIG. 5D). If the signed distance values of adjacent pixels are different and there are a plurality of candidates for the signed distance value to be substituted, a signed distance value having a small absolute value is substituted. However, the substitution method in this case is not limited to the above. For example, a signed distance value having a large absolute value may be selected, or the one substituted first may be selected. There is no particular limitation.

  As described above, if the direction-specific signed distance field expansion process is repeated for the set number of times with respect to the direction-specific signed distance fields for all directions and the generation of the direction-specific signed distance field is completed, the process of step S1400 is performed. The process ends, and the process proceeds to step S1500.

(Step S1500)
Next, in step S1500, the coordinate conversion unit 150 performs a model edge point coordinate conversion process using the position and orientation parameters input in step S1100 and the internal parameters of the imaging apparatus 10.

  Specifically, first, the three-dimensional shape model 20 is acquired from the shape model holding unit 120. Then, for all model edge points constituting the three-dimensional shape model 20, the position and direction of the model edge point expressed in the model coordinate system, the position and orientation parameters input in step S1100, and the inside of the imaging device 10 Using the parameters, the position and direction of the model edge point in the image coordinate system are calculated.

  When the coordinate conversion process for all model edge points is completed, the process of step S1500 is terminated, and the process proceeds to step S1600.

(Step S1600)
Next, in step S1600, the direction-specific reference map selection unit 160 generates a plurality of direction-specific signed distance fields generated in step S1400 based on the direction of the model edge point in the image coordinate system calculated in step S1500. A direction-specific signed distance field to be referred to in the association processing in step S1600 is selected from the inside.

First, the normal direction of the model edge point is calculated from the direction of the model edge point in the image coordinate system. Here, if the normal direction of the model edge point is (n _x , n _y ) and the direction of the signed distance field by direction is (s _x , s _y ), the normal direction of the model edge point and the direction-specific sign are added. The direction difference θ from the direction of the distance field is calculated from the following equation.

The above calculation is performed on the search direction of the plurality of direction-specific distance fields generated in step S1400, and the direction-specific distance field in the direction having the smallest direction difference θ is used as the direction-specific code for the correspondence search. Select as a distance field.

  The method for selecting the direction-specific distance field is not limited to the above. For example, when the direction difference θ is within a predetermined range for a plurality of direction-specific distance fields, a plurality of direction-specific signs are added. The distance field may be selected as the direction-by-direction signed distance field used for the correspondence search, and if the direction difference θ is larger than a certain value, there is no direction-by-direction signed distance field close to the normal direction of the model edge. There are no particular restrictions on the method and number of selection of the direction-specific signed distance field.

  The above process is performed for all model edge points, and if selection of the signed distance field by direction for each model edge point is completed, the process of step S1600 is terminated, and the process proceeds to step S1700.

(Step S1700)
Next, in step S1700, the associating unit 170 corresponds to the model edge point based on the position of the model edge point in the image coordinate system calculated in step S1500 and the signed distance field by direction selected in step S1600. The edge feature to be searched is searched, and the model edge point is associated with the edge feature.

First, by referring to the pixel corresponding to the position of the model edge point in the image coordinate system calculated in step S1500 in the direction-by-direction signed distance field selected in step S1600, the edge feature closest to the normal direction of the model edge point A signed distance value representing the distance to is acquired. Here, when the position of the model edge point in the image coordinate system is (m _x , m _y ), the obtained signed distance value is d, and the direction of the signed distance field by reference direction is (s _x , s _y ). The position (e _x , e _y ) of the edge feature closest to the model edge point in the search direction can be calculated from the following formula.

The edge feature positions (e _x , e _y ) calculated as described above are determined as the edge feature positions corresponding to the model edge points. In step S1600, when a plurality of direction-specific signed distance fields are selected for each model edge point, the position of the edge feature by each direction-specific signed distance field is calculated, and a plurality of matching results are obtained. Save the position of the edge feature. In step S1600, for the model edge point for which the direction-specific distance field is not selected, the processing in this step is skipped.

  The above processing is performed for all model edge points, and when the edge feature association processing for all model edge points is completed, this processing is terminated, and the association between the model edge point group and the edge feature group is completed. The result is determined.

(Step S1800)
Next, in step S1800, the residual calculation unit 180 calculates a residual on the image of the model edge point and the edge feature based on the pair of the model edge point and the edge feature whose correspondence is calculated in step S2100. calculate. When a plurality of edge features are associated with one model edge point, a residual in each is calculated, and a residual having a smaller residual is calculated as a residual of the model edge point. When the calculation of residuals for all model edge points is completed, the process of step S1800 is terminated, and the process proceeds to step S1900.

(Step S1900)
Next, in step S1900, the position / orientation deriving unit 190 calculates the position and orientation between the three-dimensional shape model 20 and the object to be measured based on the residual between the model edge point and the edge feature calculated in step S1800. . In this step, the position and orientation are updated by solving the linear simultaneous equations so that the residual between the model edge point and the edge feature is minimized. Since this processing is not related to the essence of the present invention, description thereof is omitted. Details are described in the following documents.

  T Drummond, R Cipolla, “Real-time visual tracking of complex structures”, Pattern Analysis and Machine Intelligence, IEEE Transactions on 243 (246).

  If the position and orientation update process is completed, the process of step S2300 is terminated, and the process proceeds to step S2000.

(Step S2000)
Next, in step S2000, the position / orientation deriving unit 190 determines whether or not the position / orientation updated in step S1900 has converged, that is, whether or not iterative calculation is required. Specifically, it is determined that convergence has occurred when the correction value is approximately 0, or when the difference before and after correction of the square sum of the error vectors is approximately 0. If not converged, the process returns to step S1800, and the residual calculation process is performed again using the updated position and orientation. If it is determined that the process has converged, this process ends.

  With the above processing, a final estimated value of the relative position and orientation between the imaging apparatus and the object to be measured is determined.

  The information processing apparatus 1 according to the present invention makes it possible to estimate the position and orientation between the imaging apparatus 10 and the measured object at high speed.

  As described above, in this embodiment, a direction-specific distance field is generated in advance for a plurality of search directions, and the direction-specific code to be referred to is determined according to the normal direction on the image of the model edge point. By changing the attached distance field, the edge feature that is nearest to the normal direction of the model edge point can be referred to at high speed. As a result, the association between the model edge point and the edge feature can be processed with high accuracy and at high speed. Then, by performing position / orientation estimation using the result, the position / orientation of the object to be measured can be estimated stably and at high speed.

[Modification 1]
In the above-described embodiment, the process of selecting the reference map for each direction based on the normal direction of the model edge point in order to search for the nearest neighbor in the normal direction of the model edge point has been described. However, the criterion for selecting the direction-specific reference map is not limited to the method described above.

  For example, in a situation where the object to be measured or the imaging device moves, in situations where the position of the object to be measured on the next frame can be roughly predicted, such as in situations where the object to be measured on the image is tracked in time series, the object The direction-specific reference map may be selected so that the nearest neighbor search is performed toward the predicted position.

  Specifically, in the direction-specific reference map selection process in step S1600 in the above-described embodiment, the direction from the current position of the model edge point of interest to the predicted position is changed to the direction instead of the normal direction of the model edge point. Used as a criterion for selecting another reference map. When the velocity vector on the model edge point image is known, the direction-specific reference map may be selected using the direction based on the velocity vector. As long as the selection is made based on the model point information, the criteria for selecting the direction-specific reference map are not limited, and the direction-specific reference map may be selected by any criteria. In this case, the image feature detected from the input image may not be an edge feature. For example, a feature point may be detected, and a direction-specific reference map indicating the feature point position may be generated and selected. In addition, when detecting feature points as image features, the 3D shape model may not be in a form having a position and direction such as a model edge point. For example, a model point that holds only a 3D position May be used as a three-dimensional shape model.

  As described above, by selecting the reference map for each direction using the prediction information of the measured object, it is possible to perform the nearest neighbor search process with higher accuracy based on the prediction information.

[Modification 2]
In the above-described embodiment, the method of using the direction-specific distance field in which the signed distance value is stored as a pixel is used as the direction-specific reference map. However, the data used as the direction-specific reference map is not limited to the above-mentioned direction-specific signed distance field. For example, an edge feature database in which an edge position and an index number (index value) are associated with each other is generated from the edge feature detected in step S1300, and each direction reference map is generated in the direction-specific reference map generation in step S1400. The index number of the edge feature that is the nearest in the predetermined search direction may be stored in the pixel. In this case, the association processing in step S1600 refers to the index number based on the model edge position in the image coordinate system, and refers to the position of the nearest edge feature via the edge feature database. As described above, the information held as the reference map for each direction may hold any information as long as the edge positions that are close in the predetermined search direction can be referred to at high speed, and are held as the reference map for each direction. There is no particular limitation on the data format.

  As described above, by maintaining the edge index as the reference map for each direction, it becomes possible to refer to the edge feature that is closest to the normal direction of the model edge point at high speed.

[Second Embodiment]
In the first embodiment, the method applied to the estimation of the position / orientation of an object in the present invention has been described. In the second embodiment, a method of applying the present invention to object matching will be described.

  FIG. 6 is a functional block diagram of the information processing apparatus 2 according to the second embodiment. Hereinafter, the same functions as those in the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  In the present embodiment, as many 3D shape models as the number of objects to be collated are held, and the direction-specific edge map selection unit 160, the association unit 170, and the residual calculation unit 180 are processed for each model, The collation unit 690 obtains a model with a small residual as a collated model.

  The shape model holding unit 620 holds a three-dimensional shape model representing the shape of an object that is a target to be collated. The method for holding the three-dimensional shape model is the same as that of the shape model holding unit 120 in the first embodiment. The difference from the first embodiment is that the present embodiment holds a plurality of three-dimensional shape models as many as the number of objects to be collated.

  The model matching unit 690 matches the feature in the image with the three-dimensional shape model. At this time, the model matching unit 690 will be described in detail later.

  Next, a processing flowchart of the second embodiment will be described. Since steps other than step 7900 are the same as those in the first embodiment, a description thereof will be omitted. However, in the present embodiment, the processes in steps S1600 to 1800 are performed by the number of models stored in the shape model storage unit 620.

(Step S7900)
Subsequently, in step S7900, the model matching unit 690 determines the three-dimensional model (three-dimensional shape model) most similar to the measurement data based on the residual obtained in step S1800 for each three-dimensional model (three-dimensional shape model). ). In this step, a median value is calculated as a statistic from the residual of the model edge in each three-dimensional shape model, and this is held as a matching degree of the three-dimensional shape model. The residual statistic is calculated for all three-dimensional shape models, and the three-dimensional shape model is collated by determining the three-dimensional shape model having the smallest residual statistic. In addition, as a statistic of a residual, for example, an average value or a mode value may be used other than the median value of the residual. Any method can be used as long as it is an index capable of determining the small residual, and there is no particular limitation.

  According to the present embodiment, it is possible to perform model matching at high speed and stably.

[Third Embodiment]
As preferred application examples of the information processing apparatuses 1 and 2 according to the present invention, the following forms are also conceivable. That is, the position and orientation of the measurement object 30 are estimated based on the grayscale image obtained by the imaging apparatus 10, and the industrial robot arm 50 is controlled based on the estimated position and orientation. An example of use in which the object to be measured is held (object holding) by the robot hand at the tip of the robot arm is given. Hereinafter, an application example of the information processing apparatus 1 according to the embodiment of the present invention will be described with reference to FIG. FIG. 7 shows a configuration example of a robot system that grips the object 30 to be measured using the information processing apparatus 1 and the robot 50.

  The robot 50 is controlled by the robot controller 40 and moves a hand to a commanded position to grip an object. Since the position of the object 30 to be measured changes on the work table, it is necessary to estimate the current position and orientation of the object 30 to be measured and to perform grip control of the robot.

  The imaging device 10 is a camera that captures a normal gray image, and is installed at a position where an object to be measured 30 such as a hand of an industrial robot arm can be imaged.

  The information processing apparatus 1 estimates the position and orientation of the measured object 30 based on the grayscale image obtained from the imaging apparatus 10. The position and orientation of the measured object 30 estimated by the information processing apparatus 1 is input to the robot 50, and the robot arm is controlled so as to grip the measured object 30 and the like. With the information processing apparatus according to the present embodiment, the robot system can hold (hold) the measured object 30 by performing position and orientation estimation even if the position of the measured object 30 is indefinite.

<Effect of embodiment>
In the first embodiment, a direction-specific signed distance field is generated in advance for a plurality of search directions, and the direction-specific signed distance field is changed according to the normal direction on the model edge point image. As a result, the edge feature that is closest to the normal direction of the model edge point can be referred to at high speed. The method of estimating the position and orientation of the object to be measured by performing model fitting so that the three-dimensional shape model representing the shape of the object to be measured is applied to the feature detected from the grayscale image has been described. By using a direction-specific reference map that can quickly refer to edge features that are nearest to each direction, the model edge points and edge features are associated with each other at high speed, and the results are used to estimate the position and orientation. It becomes possible to estimate the position and orientation of the measured object at high speed.

  In the second embodiment, the model edge point and the edge feature are associated with each other at high speed using the direction-specific reference map that can quickly refer to the edge feature that is closest to each direction, and the model is obtained using the result. By performing collation, it is possible to perform model collation at high speed and stably.

  In the third embodiment, the method of estimating the position and orientation of the object to be measured by the information processing apparatus of the present invention, and the robot system holding and moving the object to be measured based on the estimation result has been described. Thereby, even if the position of the measured object is indefinite, the robot system can hold the measured object.

<Definition>
Through the description of the specification of the present application, as a three-dimensional model information of an object to be measured, a model constituted by model edge points having information of a three-dimensional position and a three-dimensional direction has been described as an example. However, other representation methods may be used as the three-dimensional shape model. For example, a simple 3D point set, a 3D line set representing a ridgeline, a 3D surface and a polygonal shape information represented by a set of lines and 3D points, etc. May be. A three-dimensional shape model may be expressed as a set of polygons composed of three points, three sides, and one surface, or a three-dimensional shape may be expressed as a set of simple three-dimensional points. Alternatively, a method of expressing a three-dimensional shape parametrically by a combination of implicit functions may be used. As long as it conforms to the shape or characteristics of the object to be measured, there is no particular limitation on the expression method of the three-dimensional shape model.

  The image acquisition in the present specification is not limited to the input by the camera that acquires the grayscale image, and other methods may be used as long as the target image is obtained. For example, it may be a gray image or a color image. Further, the present invention is not limited to an image photographed by a camera, and there is no particular problem even if a file stored in a storage medium or an image read from an external network is input. Further, each pixel may be a distance image having depth information.

  Throughout the description of the present specification, the coordinate conversion parameter is described as an example in which the approximate position and orientation of the measured object are known in advance and the value is acquired as the coordinate conversion parameter. However, the acquisition of the coordinate conversion parameter in the present specification is not limited to this. For example, when measuring the position and orientation of an object continuously in the time axis direction, the measurement value of the previous position (previous time) may be used as the approximate position and orientation, or images are taken in various orientations. The rough position and orientation of the measured object may be estimated by holding an image of the measured object as a template and performing template matching on the input image. As long as the position and orientation of 6 degrees of freedom of the measurement object can be obtained, the input coordinate conversion parameters are not limited and may be in any format.

  As the image feature detected from the image in the present specification, any feature can be used as long as it can be associated with the three-dimensional shape model. For example, in addition to edge features, there is no particular problem even if feature points such as Harris feature points and FAST feature points are detected.

  As the reference map for each direction in this specification, as long as the reference information that can refer to the position of the edge feature in the predetermined direction is stored in each pixel constituting the reference map for each direction, the reference map for each direction is used. There is no particular limitation on the information to be generated. For example, for each pixel value, a direction-specific signed distance field that holds the signed distance to the nearest edge pixel along the direction of interest may be generated as a direction-specific reference map, or edge features detected from the image An index number for each direction may be assigned, and an index map for each direction in which an index number indicating the edge pixel closest to the target direction is held as a pixel value may be generated as a reference map for each direction. In addition, the types and number of directions for generating the direction-specific reference map are not limited to the four directions in the horizontal direction, vertical direction, diagonal upper right direction, and diagonal lower right direction on the image. A direction-specific distance field consisting only of two horizontal directions may be generated, or a total number of 10-direction direction-specific distance fields may be generated by increasing the number of diagonal divisions. As long as a direction-specific signed distance field for a plurality of directions is generated, there is no limit on the number of generated methods and directions. In addition, an image that holds a signed distance field for each direction may be generated for each direction, or a pixel value of a signed distance field for each direction is held as multi-plane information such as RGB information. The information on the signed distance field for each direction may be held as one image. As long as the information of the direction-specific signed distance field regarding a plurality of directions is held, the data format for holding the direction-specific signed distance field is not limited, and any data format may be used.

  The method for selecting the direction-specific reference map in the present specification is not limited to the method based on the normal direction of the model edge point in the image coordinate system. For example, in a situation where the position of the measured object on the image in the next frame can be roughly predicted, such as in a situation where the measured object is tracked in time series, the direction to search for the nearest neighbor toward the predicted position of the object Another reference map may be selected. As long as the selection is made based on the model point information, the criteria for selecting the direction-specific reference map are not limited, and the direction-specific reference map may be selected by any criteria.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (12)

Model holding means for holding model information of the object;
Image acquisition means for acquiring an image including the object;
Feature detection means for detecting an image feature representing the object from the image;
First reference distance information storing a distance to the detected image feature in a first direction based on a direction of model features constituting the model information, and a second direction different from the first direction Selecting means for selecting any reference distance information from the second reference distance information storing the distance to the detected image feature in
Association means for associating an image feature corresponding to the model feature based on the reference distance information selected by the selection means and the position of the model feature transformed by the coordinate transformation unit;
An information processing apparatus comprising: a derivation unit that derives the position and orientation of the object based on the result of association by the association unit.   The information according to claim 1, further comprising a generating unit that generates the first reference distance information and the second reference distance information based on the image feature detected by the feature detecting unit. Processing equipment. The first reference distance information is a first reference map having an element corresponding to a pixel of the image, and the element of the first reference map is in the first direction from the pixel corresponding to the element. Store the distance to the nearest image feature,
The second reference distance information is a second reference map having elements corresponding to the pixels of the image, and the elements of the second reference map are in the second direction from the pixels corresponding to the elements. The information processing apparatus according to claim 1, wherein a distance to the nearest image feature is stored.   4. The reference distance information according to any one of claims 1 to 3, wherein an index value of the image feature that is close to a predetermined direction is stored in each pixel constituting the map. The information processing apparatus described.   5. The information processing apparatus according to claim 1, wherein the first direction is a vertical direction of the image, and the second direction is a horizontal direction of the image. 6.   The information processing apparatus according to claim 1, wherein the distance is a signed distance. Means for obtaining the approximate position and orientation of the object;
Means for determining a direction of a model feature in the image based on the approximate position and orientation;
The information processing apparatus according to claim 1, wherein the selection unit selects the reference distance information based on a direction of a model feature in the determined image.   The information processing method according to claim 7, wherein the selection unit selects reference distance information in a direction closest to a model feature direction in the image from the first direction and the second direction. apparatus.   In addition to the first reference distance information and the second reference distance information, the selection means further detects the detected image in a third direction different from both the first direction and the second direction. Third reference distance information storing a distance to a feature, and a distance to the detected image feature in a fourth direction different from any of the first direction, the second direction, and the third direction The information processing apparatus according to claim 1, wherein any one of reference distance information is selected from the fourth reference distance information stored.   The information processing apparatus according to claim 1, wherein the feature detection unit detects an edge feature from the image. An image acquisition step of acquiring an image including an object;
A feature detection step of detecting an image feature representing the object from the image;
First reference distance information storing a distance to the detected image feature in the first direction based on the direction of the model feature constituting the model information of the object, and a second different from the first direction A selection step of selecting any reference distance information from second reference distance information storing a distance to the detected image feature in a direction;
An association step of associating an image feature corresponding to the model feature based on the reference distance information selected in the selection step and the position of the model feature transformed by the coordinate transformation unit;
An information processing method comprising a derivation step of deriving a position and orientation of the object based on a result assigned in the association step.   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1 thru | or 10.