JP2019070898A - Estimation program, estimation device and estimation method - Google Patents

Abstract

To provide a technique which reduces calculation amount needed in association processing for an image of an object and shape information on a body.SOLUTION: A computer is configured to: acquire a captured image of an object; acquire, from a storage unit, a plurality of pieces of initial attitude candidate information which associates attitude information showing attitude and the feature amount of a viewpoint image with each other, for each attitude of a body observed from any of a plurality of viewpoints; collate the feature amount of the captured image with the feature amount of the plurality of pieces of initial attitude candidate information to extract, from the plurality of pieces of initial attitude candidate information, the initial attitude candidate information including attitude information on attitude similar to the attitude of the object projected in the captured image; narrow down a segment obtained from shape information on the body based on the position of a viewpoint in which the body set in the attitude corresponding to the extracted initial attitude candidate information is observed; and estimate the position and the direction of the viewpoint to the object in three dimensional space from the image of the object by associating the segment with a feature line of the captured image.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an estimation program, an estimation device, and an estimation method.

  In recent years, systems that display images using Augmented Reality (AR) technology have become widespread. In an example of the AR technology, an object is photographed using a camera mounted on a personal computer (PC), a portable terminal device or the like, and the position and orientation of the camera in a three-dimensional space are estimated from an image of the object. Then, content information is superimposed and displayed at an arbitrary position in the image based on the determined position and orientation of the camera.

  Computer-Aided Design (CAD) data representing a three-dimensional shape of an object is used as the content information to be superimposed and displayed. In order to determine the position and orientation of the camera, at least four pairs of three-dimensional line segments of CAD data and corresponding line segments of an object in a captured image are used.

  As a first technique, there is a technique for comparing a three-dimensional structure with a model represented by model information (for example, Patent Document 1). In the first technique, the computer can select an edge line extracted from a captured image obtained by capturing an image of a three-dimensional structure by an imaging device and a ridgeline included in a model image represented by model information of the three-dimensional structure. Display with. Next, the computer receives a selection instruction indicating an edge line and a ridge line to be superimposed. Then, in response to the received selection instruction, the computer displays a superimposed image in which the model image is superimposed on the captured image such that the edge line to be superimposed and the ridge line overlap.

  Also, as a second technology, there is a texture mapping technology in which a texture including a geometric shape and a surface color or pattern is pasted when a three-dimensional object image is incorporated into a PC and used for computer graphics (for example, , Patent Document 2).

  As a third technique, there is a technique for estimating the direction of the light source at the time of shooting and the three-dimensional shape of the object from an image obtained by shooting the object using a single light source (for example, Patent Document 3).

  Further, as a fourth technique, imaging is performed by comparing the feature amount calculated from the feature points of the captured digital image with the feature amount obtained from the feature points of the document / image registered in the database. Patent Document 4 discloses a technique for searching a database for documents and images corresponding to digital images. In Patent Document 4, a search device extracts a plurality of feature points from a captured digital image. The search device determines a set of feature points local to each of the extracted feature points. The search device selects a subset of feature points from each determined set. The search device obtains invariants to the geometric transformation based on a plurality of combinations of feature points in the subset as a feature characterizing each selected subset. The search device combines the determined invariants to calculate the feature quantity. The search device casts a vote on the document / image in the database in which the corresponding feature amount is obtained in advance. Thereby, the search device searches the document / image in the database corresponding to the captured digital image.

  Furthermore, there are, for example, the techniques of Non-Patent Document 1 to Non-Patent Document 9.

JP 2017-91078 A Unexamined-Japanese-Patent No. 2003-67775 Unexamined-Japanese-Patent No. 2001-84362 WO 2006/092957 JP, 2015-118641, A

H. Uchiyama and H. Saito, “Random dot markers,” 2011 IEEE Virtual Reality Conference, pp. 35-38. R. G. Gioi et al., LSD: a Line Segment Detector, Image Processing On Line, 2 (2012), pp. 35-55. H. Uchiyama et al., “Toward augmenting everything: Detecting and tracking geometric features on planar objects,” 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 17-25, Oct. 2011. L. G. Robert et al., “Machine perception of three-dimensional solids,” MIT Lincoln Lab. Rep. TR3315, pp. 1-82, May 1963. B. G. Baumgart, “A polyhedron representation for computer vision,” Proceedings of the May 19-22, 1975, National Computer Conference and Exposition, pp. 589-596. C. Xu et al., “Pose Estimation from Line Correspondence: A Complete Analysis and a Series of Solutions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 6, pp. 1209-1222, June 2017. Z. Zhang, "A Flexible New Technique for Camera Calibration," IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, No. 11, pp. 1330-1334, Nov. 2000. Bruce G. Baumgart, "A polyhedron representation for computer vision", Proceedings of the May 19-22, 1975, National computer conference and exposition, pp. 589-596, 1975 E. Rublee et al., “ORB: an efficient alternative to SIFT or SURF,” In Proc. Of IEEE International Conference on Computer Vision, pp. 2564-2571, 2011.

  In the case of estimating the position and orientation of a camera, when the image of an object captured by a camera is associated with CAD data of the object, for example, the following process is performed in the first technique. In order to overlap an edge line extracted from a captured image with a ridgeline included in a model image represented by model information (CAD model) of the object, the user can use an initial CAD according to the posture of the object in the captured image. Adjust the model's attitude (initial model attitude).

  However, every time the posture of the object being imaged changes, when the line segment corresponding to the posture is calculated from the CAD data of the object and the corresponding line pair is searched, the line pair pair to be searched is There is no limit to the candidates of, and the load of arithmetic processing increases.

  According to one aspect of the present invention, there is provided a technique for reducing the amount of calculation involved in the process of associating an image of an object with shape information of an object.

  According to one aspect, the estimation program causes a computer to acquire a captured image obtained by capturing an object, and from the storage unit, posture information representing the posture for each posture of an object observed from any of a plurality of viewpoints And a viewpoint image representing an image of the object observed from a viewpoint corresponding to the posture information and a feature amount of the viewpoint image calculated using a plurality of feature points extracted from the viewpoint image. A plurality of initial posture candidate information which is information is acquired, the feature amount of the captured image and the feature amounts of the plurality of initial posture candidate information are collated, and the plurality of initial posture candidate information is compared based on the collation result. Initial posture candidate information including posture information of a posture similar to the posture of the target object shown in the captured image is extracted, and based on the position of the viewpoint where the object of the posture corresponding to the extracted initial posture candidate information is observed The said thing The line segment obtained from the shape information of the object is narrowed down, the narrowed line segment and the feature line detected from the captured image are associated, and based on the association result, the image in the three-dimensional space A process is performed to estimate the position and direction of the viewpoint with respect to the object.

  As one aspect of the present invention, it is possible to reduce the amount of calculation involved in the process of associating the image of the object with the shape information of the object.

An example of an image of an object is shown. It is a figure which shows the example of the initial stage attitude | position (initial model attitude | position) of the model of CAD data showing the shape of the object of FIG. FIG. 7 is a diagram showing an example of correspondence between edge lines detected from the image of FIG. 1 and outlines of the initial model posture of FIG. 2; It is a figure which shows the relationship between the attitude | position of the object in an image, and an initial model attitude | position. It is a figure which shows an example of the estimation apparatus in this embodiment. It is a figure which shows the processing flow of the whole of this embodiment. It is a figure for demonstrating the estimation process of the initial stage model in this embodiment. It is a block diagram of an estimation device in this embodiment (example 1). It is a figure which shows the example of a data structure of CAD data in this embodiment (Example 1). It is a figure which shows the flowchart of a production | generation process of CAD image DB in this embodiment (Example 1). It is a figure for demonstrating the example of the viewpoint on a regular octahedron. It is a figure which shows an example of the CAD image produced | generated from each of several initial model attitude | position candidate in this embodiment (Example 1). It is a figure which shows the example of intersection extraction from the CAD image in this embodiment (Example 1). It is a figure for demonstrating calculation of LLAH in this embodiment (Example 1). It is a figure which shows an example of CAD image DB in this embodiment (Example 1). It is a flowchart which shows the specific example of the image processing in this embodiment (Example 1). It is a figure showing the flow of initial model posture presumption processing (S24) in this embodiment (example 1). It is a figure which shows the example of the square designation | designated area | region in an image. It is a figure for demonstrating extracting the intersection of a feature line from the captured image in this embodiment (Example 1). It is a figure which shows the example of the candidate line | wire detected from the CAD data of the object reflected to the area | region of the image shown in FIG. FIG. 21 is a diagram showing an example of remaining candidate lines after removing hidden lines from the candidate lines shown in FIG. 20. It is a figure which shows the example of the candidate line showing the outer periphery of the object in this embodiment (Example 1). It is a figure which shows the example of the corresponding | compatible pair in this embodiment (Example 1). It is a figure which shows the example of the calculation method based on the area of the area | region between the projection line and the feature line in this embodiment (Example 1). It is a figure which shows the example of the calculation method based on the distance between the projection line and the feature line in this embodiment (Example 1). It is a figure which shows the example of the line segment rotated 180 degrees. It is a figure which shows the example of the candidate line which is not suitable for calculation of a parameter. It is a figure for demonstrating calculation of the error in this embodiment (Example 1). It is a figure which shows the structural example of the computer used as an estimation apparatus in this embodiment.

Hereinafter, embodiments will be described in detail with reference to the drawings.
FIG. 1 is a diagram showing an example of an image of an object. FIG. 2 is a view showing an example of an initial posture (initial model posture) of a model of CAD data representing the shape of the object in FIG. FIG. 3 is a diagram showing an example of correspondence between edge lines detected from the image of FIG. 1 and outlines of the initial model posture of FIG.

  First, as shown in FIG. 3A, an estimation device such as a portable terminal device performs edge detection processing to detect an edge line from an image. Next, as shown in FIG. 3B, the estimation device displays in the image an outline represented by CAD data of the initial model posture and a detected edge line. The user associates edge lines and outlines with one another by selecting them with a mouse or the like. In order to determine the position and orientation of the camera, it is desirable to use at least four corresponding pairs.

Next, the estimation device calculates the position and orientation of the camera using the combination of the edge line and the contour line associated with each other. Then, as shown in FIG. 5C, the estimation apparatus superimposes and displays the outline represented by the CAD data on the image of the object in accordance with the calculated position and orientation of the camera.
According to such an association method, it is considered that the following problems occur.

  FIG. 4 is a view showing the relationship between the posture of an object in an image and the initial model posture. The posture of the CAD data model is determined according to the position of the camera that captures the object represented by the CAD data.

  In FIG. 3B, the user first sets the initial model posture 101 to a fixed value, and moves the position of the camera. Next, the user moves the initial model posture 101 by a touch operation, a mouse operation, or the like so as to match the posture 102 of the object. At this time, when the posture 102 of the object matches the initial model posture 101, the edge line and the outline can be associated (FIG. 4A). On the other hand, if the posture 102 of the object and the initial model posture 101 do not match, the edge line and the contour can not be correlated (FIG. 4B).

  Therefore, in the first technique, the initial model posture 101 is set according to the posture 102 of the object in the image in the correspondence between the edge line extracted from the captured image and the outline (line segment) represented by the CAD data. I have adjusted manually.

  However, it takes time and effort to manually adjust the initial model posture in accordance with the posture of the object in the captured image. On the other hand, if the adjustment is not performed, when forming a line segment pair, there is a possibility that a line segment pair is formed by incorrect association.

  Also, every time the posture of the object being imaged changes, if the line segment corresponding to the posture is calculated from the CAD data of the object and the corresponding line pair is searched, the line edge pair to be searched is There is no limit to the candidates to be searched, and the burden of arithmetic processing becomes large.

  Therefore, in the present embodiment, an initial model posture of CAD data of the object is estimated from an image obtained by imaging the object. Thereby, the correspondence between the edge line extracted from the captured image and the outline (line segment) represented by the CAD data is simplified, thereby reducing the time of the correspondence, improving the usability, and preventing the erroneous superimposed display. To achieve.

  FIG. 5 is a diagram showing an example of the estimation apparatus in the present embodiment. The estimation device 1 includes a storage unit 2, an image acquisition unit 3, a collation unit 4, a narrowing unit 5, an association unit 6, and an estimation unit 7. As an example of the estimation device 1, an estimation device 21 described later can be mentioned.

  The storage unit 2 stores a plurality of initial posture candidate information 9. An example of the storage unit 2 is a storage unit 33 described later. The initial posture candidate information 9 is information in which posture information, a viewpoint image, and a feature amount of a viewpoint image are associated with each posture of an object observed from any of a plurality of viewpoints. The posture information represents the posture of the object. The viewpoint image indicates the image of the object observed from the viewpoint corresponding to the posture information. The feature amount of the posture image is calculated using a plurality of feature points extracted from the viewpoint image. An example of the initial posture candidate information 9 is a CAD image DB 11 described later.

  The image acquisition unit 3 acquires a captured image obtained by capturing an object. An example of the image acquisition unit 3 is an image acquisition unit 25 described later.

  The collation unit 4 collates the feature amount of the captured image with the feature amounts of the plurality of initial posture candidate information, and based on the comparison result, from the plurality of initial posture candidate information 9 to the posture of the object shown in the captured image Initial posture candidate information including posture information of similar postures is extracted. As an example of the matching unit 4, a posture estimation unit 27 described later can be mentioned.

  The narrowing-down unit 5 narrows down the line segment obtained from the shape information of the object based on the position of the viewpoint at which the object in the posture corresponding to the initial posture candidate information extracted by the collation unit 4 is observed. An example of the narrowing-down unit 5 includes a candidate line extraction unit 24 described later.

  The associating unit 6 associates the narrowed line segment with the feature line detected from the captured image. An example of the association unit 6 is a generation unit 28 described later.

  The estimation unit 7 estimates the position and the direction of the viewpoint with respect to the object in the three-dimensional space from the image of the object based on the association result.

  By configuring in this manner, it is possible to reduce the amount of calculation involved in the process of associating the image of the object with the shape information of the object.

  The initial posture candidate information 9 is information on the posture of the object observed from each of the viewpoints when the viewpoint is arranged at the center of the apex and surface of the polyhedron surrounding the object. The number of initial posture candidate information 9 corresponds to the number of vertexes of the polyhedron and the center of the surface.

  With this configuration, when the number of viewpoints increases as the number of vertices and faces of the polyhedron increases, the initial model posture candidates also increase and the accuracy increases, and the intervals of the positions of the viewpoints to be imaged can be made uniform. .

  The candidate line extraction unit 24 further includes a generation unit 8. The generation unit 8 generates, based on the shape information of the object, a plurality of pieces of initial posture candidate information related to the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the vertices of the polyhedron surrounding the object and the center of the surface. .

  By configuring in this way, it is possible to generate initial posture candidate information for each different posture from the shape information of the object.

  The feature amount is a feature amount of each intersection obtained by calculating an invariant to a geometrical transformation with respect to a plurality of intersections around each of the intersections of the feature lines extracted from the viewpoint image or the captured image. . The matching unit 4 measures the number of intersections at which the feature amount at the intersection of the feature lines extracted from the viewpoint image matches the feature amount at the intersections of the feature lines extracted from the captured image. Information is estimated as initial attitude information.

  By configuring in this manner, the feature amount of the image can be calculated, and therefore, it is possible to estimate the initial posture information of the object in the captured image based on the feature amount.

  The posture information is represented by a three-dimensional coordinate system including a rotation component. As described later, the posture information may include at least a rotation component.

  Here, the above-mentioned feature quantities will be described in detail. In order to obtain feature quantities of the image that do not depend on distortion of the geometric transformation, feature quantities are calculated using invariants for the geometric transformation. "Invariant" is a generic term for quantities that do not change even after geometric transformation. An example of geometrical transformation is rotation, and rotating the image does not change the area of the figure shown in the image. The area of a figure is an example of an invariant to rotation. Also, the ratio of side lengths is an example of an invariant to scaling. Geometric transformations include projective transformations and affine transformations as well as similarity transformations such as rotation and scaling.

  In this embodiment, a multiple ratio is used as an example of the invariant. A multiple ratio is a value obtained from four points on a straight line or five points on a plane, and is known as an invariant to projective transformation which is a type of geometrical transformation. In order to use a multiple ratio, the feature of the target image is represented by a point (feature point).

  It is not realistic to try all possible correspondences of feature points, as combination explosions occur. Therefore, in the present embodiment, a vote using a hash is introduced, and a search is performed without explicitly taking the correspondence of feature points. At the time of registration, first, feature points are obtained from the image and feature amounts are calculated, and an index is obtained therefrom to be registered in the hash. At the time of search, the hash value is obtained by obtaining the feature point, the feature amount, and the index in the same process from the search query, and the search is performed by voting on the registered document / image.

  Note that “voting” is a process used to partially accumulate evidence in the field of information processing, and based on the obtained evidence, any one of the options is scored and all evidence is aggregated Result It is the process of selecting the option with the highest score. In general, each piece of evidence has a different score. Also, “search query” refers to data representing a user's search request in information search. In the case of this embodiment, the user inputs an image as a search question. That is, the user inputs an image in the sense that "the same as this image can be retrieved from the database".

  In the present embodiment, focusing on a certain point, feature quantities are defined in the vicinity of that point. That is, n points near a certain point are taken out. Furthermore, if m points (the number of points in multiple units, 4 or 5 <m <n) are extracted from n points, n points near corresponding points are obtained even if geometrical transformation is performed. It is considered that there exists a combination such that m points in the above match. In this case, all combinations of m points out of n neighborhoods are tried for all points. By appropriately selecting the values of n and m, explosion of computational complexity can be avoided.

  Furthermore, if you select 4 or 5 points from m points and calculate the ratio, it is very rare that all ratios will match even if some ratios are the same as other images. become. As a result, identity can be determined with high accuracy.

  FIG. 6 is a diagram showing an overall processing flow of the present embodiment. FIG. 7 is a diagram for explaining an estimation process of an initial model in the present embodiment. First, the estimation apparatus 1 generates a CAD image database (hereinafter, the database is referred to as “DB”) 11 in which candidate data of an initial model posture is stored based on CAD data of an object (S1).

  The CAD image DB 11 includes an image (CAD image) of a CAD model representing the posture of the object observed from each viewpoint when the object is observed from a plurality of viewpoints, and the position and orientation (Ri: rotation component) of the CAD model at that time , Ti: translational component) is stored. Here, the viewpoint corresponds to the above-described camera.

  The CAD image DB 11 further stores feature amounts f for specifying candidates (initial model posture candidates) of postures of the CAD model in the CAD image. Here, the estimation device 1 calculates, for example, an intersection point of straight lines obtained by extending the outline of the object from the CAD image. Then, the estimation apparatus 1 determines n intersections in the vicinity of each intersection p as a local set, and calculates the feature amount f for each combination for selecting m intersections from each set.

  Next, the estimation device 1 acquires a captured image 12 in which an object is captured (S2). The estimation device 1 performs initial model attitude estimation processing (S3). In S3, the estimation device 1 searches the CAD image DB 11 for a CAD image closest to the posture of the object in the captured image 12, and determines the model position and posture in the CAD image as the initial model posture.

  In S3, the estimation apparatus 1 first performs matching between the feature quantities of the respective initial model posture candidates stored in the CAD image DB 11 and the feature quantities of the image portion of the target object included in the captured image (S3-1) . Here, the estimation device 1 calculates, for example, an intersection of straight lines obtained by extending the outline of the object from the captured image. Then, the estimation apparatus 1 determines n intersections in the vicinity of each intersection p as a local set, and calculates the feature amount f for each combination for selecting m intersections from each set. The estimation apparatus 1 measures the number of intersections at which the feature amounts match between the captured image and the initial model posture candidate.

  The estimation device 1 calculates the number of matching intersection points as the degree of similarity. From the initial model posture candidates stored in the CAD image DB 11, the estimation apparatus 1 determines, as the initial model posture 13, one having the highest similarity to the posture of the object included in the captured image (S3-2).

  The estimation apparatus 1 extracts a plurality of candidate lines observed from the viewpoint of the observation of the object among the plurality of contour lines (candidate lines) included in the shape information of the object of the initial model posture 13 (S4).

  Next, the estimation apparatus 1 determines a predetermined number of candidate lines to be observed, each of which is associated with a predetermined number of feature lines detected from the image. A plurality of pieces of association information indicating combinations are generated (S5). Then, the estimation device 1 determines the association result based on the errors of each of the plurality of association information (S6).

  The estimation apparatus 1 estimates the position and direction (the position and orientation of the camera) of the viewpoint with respect to the object in the three-dimensional space from the image of the object based on the association result (S7).

The present embodiment will be described in detail below.
FIG. 8 is a block diagram of an estimation apparatus in the present embodiment (Example 1). The estimation device 21 includes a CAD data reading unit 22, a CAD image DB generation unit 23, a candidate line extraction unit 24, an image acquisition unit 25, a feature detection unit 26, a posture estimation unit 27, a generation unit 28, a position calculation unit 29, and an error calculation. A unit 30, a determination unit 31, an output unit 32, and a storage unit 33 are included.

  The estimation device 21 may be a portable terminal device such as a tablet, a notebook PC, or a smart device, or may be a computer such as a desktop PC.

  The storage unit 33 stores the captured image 12, CAD data 41, CAD image DB 11, feature line 43, candidate line 44, corresponding pair 45, parameter 46, index 47, parameter 48 and the like.

  The CAD data 41 includes vertex information of a plurality of vertices representing a three-dimensional shape of an object, and line segment information of a plurality of line segments. The vertex information includes three-dimensional coordinates of each vertex of the object, and the line segment information includes identification information indicating vertices at both ends of each line segment.

  The CAD image DB generation unit 23 determines a candidate (initial model posture candidate) of a value (posture value) indicating the position and posture of the object when the object represented by the CAD data 41 is observed from each of a plurality of imaging positions. . The CAD image DB generation unit 23 generates an image (CAD image) of the image of the CAD object of each of the determined initial model posture candidates using the rendering program. The CAD image DB generation unit 23 extracts an edge (feature line) from the generated CAD image, extracts an intersection of straight lines extending the edge (feature line), and calculates a feature amount of each intersection. The CAD image DB generation unit 23 associates the posture value (Ri, Ti) of the initial model posture candidate, the CAD image corresponding to the posture value, and the feature amount of each intersection corresponding to the CAD image, It stores in image DB11.

  The imaging device 20 is, for example, a camera, and captures a captured image 12 of an object. The image acquisition unit 25 acquires the captured image 12 from the imaging device 20 and stores the acquired image 12 in the storage unit 33.

  The feature detection unit 26 performs edge detection processing, detects a plurality of edge lines from the captured image 12, and stores the detected edge lines in the storage unit 33 as feature lines 43.

  The posture estimation unit 27 determines, among the initial model posture candidates stored in the CAD image DB 11, an initial model posture candidate closest to the object in the captured image 12 as an initial model posture.

  The candidate line extraction unit 24 extracts the outline of the object in the determined initial model posture, and stores the outline as a candidate line 44 in the storage unit 33.

  The generation unit 28 extracts, from among the feature lines 43 detected from the captured image 12, a plurality of feature lines that satisfy a predetermined condition. As the predetermined condition, for example, it is used that at least a part of the feature line is included in a designated area in the captured image 12, that the feature line is within a range of a predetermined distance from a reference position in the captured image 12, or the like. . Further, the generation unit 28 extracts, from the candidate lines 44 detected from the initial model posture, a plurality of candidate lines observed from the position and the posture of the viewpoint at which the initial model posture is observed.

  Next, the generation unit 28 associates the N candidate lines (N is an integer of 2 or more) of the extracted candidate lines with the N feature lines of the extracted feature lines. Generate a combination of Then, the generation unit 28 stores the generated N combinations as N corresponding pairs 45 in the storage unit 33.

  The position calculation unit 29 calculates the position and orientation of the imaging device 20 when capturing the captured image 12 using the N corresponding pairs 45, and stores the calculated position and orientation in the storage unit 33 as the parameter 46. .

  At this time, the position calculation unit 29 generates a projection line by projecting a candidate line included in each corresponding pair onto the captured image 12 while changing a variable representing the position and orientation of the imaging device 20 by a predetermined value. Do.

  The error calculation unit 30 calculates an error representing a deviation between the position of the projection line generated by the position calculation unit 29 and the position of the feature line included in the corresponding pair. Then, the position calculation unit 29 obtains, as the parameter 46, the value of the variable that minimizes the total sum of the errors calculated by the error calculation unit 30.

  The position calculation unit 29 repeats the process of calculating the parameter 46 a plurality of times while changing the selection of the N corresponding pairs.

  The determination unit 31 captures a candidate line included in the N corresponding pairs selected by the position calculation unit 29 using the position and orientation of the imaging device 20 represented by the parameter 46 each time the parameter 46 is calculated. By projecting onto 12, N projection lines are generated. Then, the determination unit 31 calculates the sum of the errors between the positions of the N projection lines and the positions of the N feature lines in the same manner as the error calculation unit 30, and calculates the sum of the calculated errors as the index 47. Are stored in the storage unit 33 as

  Next, the determination unit 31 determines N corresponding pairs that minimize the sum of errors based on the indices 47 calculated using the respective parameters 46. These N corresponding pairs correspond to the association results. Then, using the determined N corresponding pairs, the determination unit 31 calculates the position and orientation of the imaging device 20 in the three-dimensional space, and stores the calculated position and orientation in the storage unit 33 as the parameter 48. The output unit 32 outputs the parameter 48 as a processing result.

  FIG. 9 is a view showing an example of the data structure of CAD data in the present embodiment (Example 1). The CAD data 41 is, for example, data of a CAD object in OBJ format. The CAD data 41 stores, for example, CAD data structures represented by line segments forming a CAD object, that is, the start point and the end point of a three-dimensional straight line. The CAD data structure in FIG. 9 includes “straight line number” 41-1 for identifying a straight line, “start point [mm]” 41-2 indicating a starting point coordinate of the straight line, and “end point [mm]” 41 Including 3

  FIG. 10 is a diagram showing a flowchart of CAD image DB generation processing in the present embodiment (Example 1). FIG. 11 is a diagram for explaining an example of a viewpoint on a regular octahedron. FIG. 12 is a view showing an example of a CAD image generated from each of a plurality of initial model posture candidates in the present embodiment (Example 1). FIG. 13 is a view showing an example of extracting an intersection point from a CAD image in the present embodiment (Example 1).

The CAD data reading unit 22 reads the CAD data 41 and stores it in the storage unit 33 (S11). Here, the variable i is set to 0 (S12)
The CAD image DB generation unit 23 determines the position and orientation (Ri, Ti) of the initial model orientation candidate Ki of the CAD object based on the CAD data 41 (S13). In S13, the CAD image DB generation unit 23 determines the initial model posture candidate Ki from any of the positions and orientations of the viewpoint that points from the vertex of the polyhedron surrounding the CAD object and the center of the surface to the CAD object. For example, the CAD image DB generation unit 23 sets the CAD object at the center of the regular octahedron (FIG. 11), and selects the initial model from any of the positions and orientations of viewpoints facing the CAD object from the vertex of the regular octahedron and the center of the surface. Calculate the posture candidate Ki.

  In addition, not a regular octahedron but another regular polyhedron can also be used as a polyhedron which surrounds a CAD object. As the number of viewpoints increases, the initial model posture candidates increase and the accuracy increases, but the capacity of the CAD image DB 11 increases and the amount of calculation processing also increases. Also, the reason for using regular polyhedrons is to make the camera spacing even.

  The CAD image DB generation unit 23 is a rendering program such as OpenGL, and draws an image (CAD image) of a CAD object at the position and orientation (Ri, Ti) of the initial model orientation candidate Ki (S14). An example of a CAD image drawn based on the position and orientation (Ri, Ti) (for example, i = 1 to 6) of the initial model orientation candidate Ki is shown in FIG.

  The CAD image DB generation unit 23 extracts an edge of the CAD image generated for the position and orientation (Ri, Ti) of the initial model orientation candidate Ki, and extracts an intersection of straight lines extending the edge (S15). In S15, the CAD image DB generation unit 23 performs edge extraction processing on the CAD image generated in S14. As the edge extraction processing, for example, it is conceivable to use the processing described in Non-Patent Document 2. As shown in FIG. 13, the CAD image DB generation unit 23 extends the extracted edge, and extracts a point intersecting with another edge as an intersection point. Here, the reason for extending the edge is that the edge may be shredded when performing the same intersection extraction on the actual captured image.

  The CAD image DB generation unit 23 calculates a feature quantity (hereinafter, referred to as “LLAH feature quantity”) fi based on, for example, Locally Likely Arrangement Hashing (LLAH) for each of the intersections extracted in S15 (S16). The LLAH uses a combination of positional relationships with a plurality of feature points existing in the vicinity of a certain feature point, and can correspond to a certain degree of viewpoint change and loss of an intersection point. LLAH is a feature defined for the distribution of points.

  The CAD image DB generation unit 23 determines n intersections in the vicinity of each intersection p as a local set, and calculates a feature amount for each combination for selecting m intersections from each set. Here, in order to obtain feature quantities of a shape specified by an edge in an image that does not depend on distortion of geometric transformation, feature quantities are calculated using invariants for geometric transformation. In the present embodiment, a multiple ratio is taken as an example of an invariant. A multiple ratio is a value obtained from four points on a straight line or five points on a plane, and is known as an invariant to projective transformation which is a type of geometrical transformation.

Here, the candidate ID is an identification number that specifies the posture of the object. The point ID is an identification number of a point assigned to each point for each document. The nCm pattern ID is an identification number given to the combination pattern when extracting m points from n points, and takes a value of 0 to Cm-1. Similarly mC ₅ pattern ID is an identification number of a combination pattern when extracting five points from m point takes a value mC ₅ -1 0.

  For example, the case where 5 points are taken out from m (= 5 points) will be described as an example. In this case, five to five multiple ratios are calculated. As a multiple ratio obtained from five points on the same plane, for example, there are the following.

Here, P (A, B, C) is the area of a triangle composed of vertices A, B, C. In the present embodiment, for example, feature quantities unique to an image are calculated using such a multiple ratio, and an image search is performed.

  The five multiple ratios are obtained by circulating the leading points such as ABCDE, BCDEA, CDEAB, DEABC, and EABCD with respect to the obtained five-point ABCDE.

  Next, the index of the hash table is calculated. The hash function is shown below.

Here, cr _n (n = 0 to 4) is five discretized multiple ratios, Vmax is a maximum value of the discretized multiple ratios, and pat is an _m C ₅ pattern ID.

After that, a set of (candidate ID, intersection ID, _n C _m pattern ID) is registered in the hash table H 1 using the index. If a collision occurs in the hash, data is added in a list structure. Here, not only the candidate ID but also the intersection ID and _n C _m pattern ID are registered in the hash table H 1 when the feature quantities are compared at the time of the search. , Intersection ID, and _n C _m pattern ID.

FIG. 14 is a diagram for explaining the calculation of LLAH in the present embodiment (Example 1). In the example of FIG. 14, the area ratio (A1 / A2) of triangle pairs existing around a certain point p is calculated as a feature amount (for example, see Non-Patent Document 3). In FIG. 14, the feature quantity of the point p is represented by a numeric string of ₇ C _m in number and _m C ₄ in dimension, where n (= 7) is a number of points nearby, for example. It is assumed that m is the number of feature points to be used out of n, for example, five.

  In this way, if you select 4 or 5 points from m points and calculate the ratio, even if some ratios are the same as other images, it is very possible that all ratios match. Become rare. As a result, identity can be determined with high accuracy.

  The CAD image DB generation unit 23 causes the CAD image DB 11 to include the position and orientation (Ri, Ti) of the initial model orientation candidate Ki, the CAD image corresponding to the initial model orientation candidate, and the LLAH feature value for each intersection included in the CAD image Store fi (S17). The CAD image DB 11 will be described with reference to FIG.

  FIG. 15 shows an example of a CAD image DB in the present embodiment (Example 1). The CAD image DB 11 includes data items of “ID” 11-1, “rotational component Ri” 11-2, “translational component Ti” 11-3, “LLAH feature amount fi” 11-4, and “CAD image” 11-5. including.

In the "ID" 11-1, identification information for identifying the initial model posture candidate Ki is stored. In the example of the present embodiment, ID = i. The rotational component Ri (Ri_x, Ri_y, Ri_z) of the initial model posture candidate Ki specified by ID = i is stored in the “rotational component Ri” 11-2. The translation component Ti (Ti_x, Ti_y, Ti_z) of the initial model posture candidate Ki specified by ID = i is stored in the “translation component Ti” 11-3. In the “LLAH feature quantity fi” 11-4, the LLAH feature quantity fi at each intersection point extracted from the initial model posture candidate specified by ID = i is stored. When m = 5, for example, ₇ C ₅ × ₅ C ₄ × (number of intersections) LLHA feature quantities fi are obtained for each ID. In the “CAD image” 11-5, a CAD image of an initial model posture candidate Ki specified by ID = i is stored.

  After the process of S17, the CAD image DB generation unit 23 increments i (S18). The CAD image DB generation unit 23 repeats the processing of S13 to S18 while i is equal to or less than the predetermined initial model posture candidate number J (S19, NO). The predetermined initial model posture candidate number J is, for example, the total number of the vertexes of the polyhedron described in S13 and the center of the plane. When i exceeds the predetermined initial model posture candidate number J (S19, YES), the flow of FIG. 10 ends.

  Thus, the CAD image DB 11 is generated in advance. Next, estimation processing including processing for estimating an initial model posture corresponding to an object posture of a captured image from the captured image 12 will be described.

  FIG. 16 is a flowchart showing a specific example of the image processing in the present embodiment (Example 1). The CAD data reading unit 22 reads the CAD data 41 (S20). The candidate line extraction unit 24 detects a three-dimensional straight line from the CAD data 41, and creates a data structure as shown in FIG. 9 (S21).

  The image acquisition unit 25 acquires the captured image 12 from the imaging device 20 (S22), and the feature detection unit 26 detects a plurality of feature lines 43 from the captured image 12 (S23).

  Next, the posture estimation unit 27 estimates an initial model posture corresponding to the posture of the object in the captured image 12 using the CAD image DB 11 (S24). Details of S24 will be described with reference to FIG.

  FIG. 17 is a diagram showing a flow of initial model posture estimation processing (S24) in the present embodiment (Example 1). First, the user designates upper left coordinates and lower right coordinates in the image by using a quadrilateral designated area that designates the presence range of the object in the captured image 12 (S24-1). Then, the posture estimation unit 27 extracts an edge (feature line) of an image portion indicating an object within the designated region designated. S24-1 will be described with reference to FIG.

  FIG. 18 is a diagram illustrating an example of a rectangular designated area in the captured image 12. When the rectangular designated area 51 is designated by the user in the captured image 12, the feature detection unit 26 extracts a feature line partially or entirely included in the designated area 51.

  FIG. 19 is a diagram for describing extraction of intersections of feature lines from a captured image in the present embodiment (Example 1). As shown in FIG. 19, the posture estimation unit 27 extends each edge extracted in the designated area 51, and extracts an intersection of the edges (S24-2). The process of S24-2 is the same process as the process described in S15 and FIG.

  The posture estimation unit 27 calculates the LLAH feature quantity fq for each intersection point (S24-3). The process of S24-3 is the same process as the process described in S16 and FIG. The posture estimation unit 27 obtains a hash index in the hash table by the method described in S16, and obtains a similar hash table H2.

  Posture estimation unit 27 is similar to the initial model posture candidate stored in CAD image DB 11 by the matching of each intersection point using LLAH feature quantity fi and LLAH feature quantity fq, and the one with the largest number of intersection points where the feature quantity matches is similar It identifies as a candidate whose degree becomes the largest (S24-4).

  In S24-4, the posture estimation unit 27 searches for an intersection point in the CAD image corresponding to the intersection point in the captured image for one captured image 12 and one CAD image. The posture estimation unit 27 stores the number of corresponding intersection points, and repeats the process with other CAD images (see Patent Document 4).

More specifically, posture estimation unit 27 uses H1 and H2 to check whether or not multiple ratios equal to or greater than a certain number L match, including the order, and measure the number of matching points for the matched items. . In order to determine a suitable value of L, a preliminary experiment is performed by selecting a plurality of appropriate values under the constraint of _m C _n , and the value of the ratio between the correct answer and the incorrect answer is large. It may be determined as the value of L.

  By the above-described process, the posture estimation unit 27 specifies an initial model specification candidate corresponding to the candidate ID with the largest number of the finally matching intersection points as a candidate with the highest degree of similarity.

  The posture estimation unit 27 determines an initial model posture candidate of the CAD image having the largest number of corresponding intersection points as the initial model posture (S24-5).

  It returns to the explanation of FIG. After the end of the initial model posture estimation process (S24), the candidate line extraction unit 24 extracts candidate lines from the determined initial model posture (S25). In S25, first, the candidate line extraction unit 24 coordinate-converts CAD data in accordance with the selected initial model posture. Here, the three-dimensional straight line detected in S21 is used. The candidate line extraction unit 24 removes lines (hidden lines) present on the back side of the viewpoint among the three-dimensional straight lines from the coordinate information of the CAD data by the method described in Non-Patent Document 4. The candidate line extraction unit 24 extracts a line (peripheral line) existing at the outer periphery of the three-dimensional straight line from the coordinate information of the CAD data. Here, there may be a case where a line other than the hidden line is used, and a case where only the outer peripheral line is used. S25 will be described in detail with reference to FIGS.

  FIG. 20 is a view showing an example of the candidate line 44 detected from the CAD data 41 of the object shown in the area 51 of the captured image 12 shown in FIG. In this example, 25 candidate lines 44 are detected.

  FIG. 21 is a diagram showing an example of the remaining candidate lines after removing the hidden lines from the candidate lines 44 shown in FIG. By removing 11 hidden lines out of the 25 candidate lines 44, 14 candidate lines are extracted.

  The candidate line extraction unit 24 may extract a candidate line representing the outer edge of the object among the remaining candidate lines, and further extract a candidate line representing the outer periphery of the object among candidate lines representing the outer edge. Good. The candidate line representing the outer periphery of the object represents the outline of the object, and can be detected by using the technique of Boundary Representations in computer graphics (CG).

  For example, as disclosed in Non-Patent Document 8, data of a Winged-Edge (Winged-Egde) structure, which is one of boundary representations, includes information representing an outline and vertices and faces forming the outline; And information representing the connection with other contours. Based on these pieces of information, it is possible to determine whether each candidate line detected from the CAD data corresponds to the outer circumference.

  When the feature line 43 detected from the captured image 12 is an outline, a boundary between the object and the background is detected as the feature line 43. Since the object and the background are physically separated, the light such as the sun or the illumination may be different in how it is hit, or in different materials or colors in many cases. For this reason, a clearer feature line is easily detected, and the accuracy of the position of the feature line is also enhanced. In addition, by generating a large number of corresponding pairs representing the outline, the distribution range of the corresponding pairs in the captured image 12 is broadened, which is considered to contribute to the improvement of the calculation accuracy of the parameter 48.

  FIG. 22 is a diagram showing an example of candidate lines representing the outer periphery of an object in the present embodiment (Example 1). Eight candidate lines indicated by thick lines in FIG. 22 are extracted as candidate lines representing the outer periphery.

  The candidate line extraction unit 24 may select and use a candidate line longer than a predetermined length when projected onto the captured image 12 among the candidate lines shown in FIGS. When the projection line is long, since the outline representing the shape of the object itself is long, it is likely to be associated with a longer feature line. Also, it is considered that the longer the feature line, the higher its reliability. Furthermore, in the calculation of the position and orientation of the imaging device 20, as the projection line and the feature line are both longer, the calculation accuracy of the error between the projection line and the feature line is improved, so that the calculation accuracy of the parameter 48 is also improved.

  Next, the generation unit 28 generates N corresponding pairs 45 in which the N candidate lines and the N feature lines are associated (S26). Here, the generation unit 28 selects a straight line of the corresponding pair. The generation unit 28 randomly selects four edges from four edges in the image and candidate straight lines of CAD data. The generation unit 28 may sort the line segments in the long order, and may preferentially select the long line. This improves the stability of later attitude estimation.

  The position calculation unit 29 calculates the position and orientation (parameter 46) of the imaging device 20 using the generated corresponding pair 45 (S27). The position calculation unit 29 calculates the position and orientation (R ′, T ′) of the imaging device 20 with respect to the model by, for example, the method described in Non-Patent Document 6.

  FIG. 23 is a view showing an example of corresponding pairs in the present embodiment (Example 1). In this example, four corresponding pairs are generated by associating candidate lines 1711 to 1714 with feature lines 1701 to 1704, respectively.

  In S27, the position calculation unit 29 can calculate the parameter 46 using, for example, the least squares method. In this case, the position calculation unit 29 generates a projection line by projecting a candidate line included in each corresponding pair onto the captured image 12 while changing a variable representing the position and orientation of the imaging device 20 by a predetermined value. Do.

  The error calculation unit 30 evaluates an error Ei (i = 1 to N) between the position of the projection line and the position of the feature line included in the corresponding pair, and the sum E of squared errors for N corresponding pairs is minimum. The value of the variable which becomes The sum E of squared errors is calculated by the following equation.

  The error calculation unit 30 can calculate the error Ei by, for example, a method as shown in FIG. 24 or 25. FIG. 24 is a diagram showing an example of a calculation method based on the area of the region between the projection line and the feature line in the present embodiment (Example 1). When the projection line included in the i-th corresponding pair is the line segment 1801 and the feature line is the line segment 1802, the line segment 1803 and the line segment 1804 connecting the both ends of the line segment 1801 and the both ends of the line segment 1802 are It can be defined. In this case, the area Ai of the region surrounded by the line segments 1801 to 1804 can be used as the error Ei (Ei = Ai).

  As the area Ai is smaller, the error Ei is smaller. When the line segment 1801 overlaps the line segment 1802, the error Ei is zero.

  FIG. 25 is a diagram showing an example of a calculation method based on the distance between the projection line and the feature line in the present embodiment (Example 1). Let the lengths of the perpendicular 1901 and the perpendicular 1902 dropped onto the line segment 1801 from both ends of the line segment 1802 be Li1 and Li2, respectively. In this case, the sum of Li1 and Li2 can be used as the error Ei (Ei = Li1 + Li2).

  As Li1 and Li2 become shorter, the error Ei becomes smaller, and when the line segment 1801 overlaps the line segment 1802, the error Ei becomes zero.

  Next, the determination unit 31 projects the N projection lines by projecting the candidate lines included in the N corresponding pairs onto the captured image 12 using the position and orientation of the imaging device 20 represented by the parameter 46. It generates (S28).

  The determination unit 31 projects a line segment of CAD data into a two-dimensional image plane according to the following equation.

(X, Y, Z): Three-dimensional coordinates A of the end point of the CAD line segment: Internal parameters of the camera (previously measured by the method of Non-Patent Document 7)
R: The calculated R 'converted to a 3-by-3 matrix by the Rodrigues' rotation formula T: The calculated T'
(U, v): 2D coordinates of end points of CAD line segment projected in image plane

  Next, the determination unit 31 calculates an index 47 representing the sum of errors between the positions of the N projection lines and the positions of the N feature lines (S29), and the index 47 is calculated a predetermined number of times It is checked whether or not it is (S30). When the index 47 has not been calculated a predetermined number of times (S30, NO), the position calculation unit 29 changes the selection of N corresponding pairs (S26), and the estimation device 21 repeats the processes after S27.

  When the index 47 has been calculated a predetermined number of times (S30, YES), the determination unit 31 selects N corresponding pairs that minimize the sum of errors (S31), and the parameter 48 is selected based on those corresponding pairs. Calculate (S32). Then, the output unit 32 outputs the selected N corresponding pairs and the parameter 48 (S33).

  According to the image processing of FIG. 16, by repeating the calculation of the index 48 while automatically changing the selection of the N corresponding pairs, it is possible to obtain the N corresponding pairs that minimize the total error. As a result, the working time of the selection operation by the user is reduced, the processing time is shortened, and the estimation accuracy of the position and orientation of the imaging device 20 is improved.

  In addition, since there is no selection error due to human error, there is no increase in processing time due to reselection. Since it is possible to determine the optimal N corresponding pairs without being a skilled person, it is possible to expand the types of work to which the matching result is applied and the target person.

  In S30 of FIG. 16, the estimation device 21 terminates the iterative process when the error represented by the indicator 47 becomes smaller than a predetermined value, instead of terminating the process repeatedly when the index 47 is calculated a predetermined number of times. May be

  In addition, in steps S27 and S29, the estimation device 21 may evaluate the similarity between the projection line and the feature line instead of the error between the position of the projection line and the position of the feature line. As the similarity between the projection line and the feature line, for example, the similarity between two line segments described in Patent Document 5 can be used. In this case, in S27, the parameter 46 that maximizes the sum of the similarities is obtained, and in S31, N corresponding pairs that maximize the sum of the similarities are selected.

  By the way, even if the sum of the errors of the N corresponding pairs selected in S31 is minimum, each projection line may represent a line segment obtained by rotating each feature line by 180 degrees.

  FIG. 26 is a diagram showing an example of a line segment rotated 180 degrees. Of the projection line and the feature line in FIG. 26, the projection line 2012 overlaps the feature line 2002. On the other hand, the projection line 2011, the projection line 2013, and the projection line 2014 correspond to line segments obtained by rotating the feature line 2002, the feature line 2003, and the feature line 2004 by 180 degrees about the projection line 2012 as an axis. In this case, since the area Ai of equation (12) is approximately zero, the sum of errors may be minimized.

Therefore, in order to prohibit such correspondence, the determination unit 31 selects N corresponding pairs that minimize the sum of errors from among the N corresponding pairs that satisfy the following conditions: It is also good.
(C11) Of the N projection lines, a predetermined proportion of projection lines are included in the image 821.
(C12) Among the N projection lines, a projection line of a predetermined ratio exists in the vicinity of a predetermined position in the image 821.
(C13) The ratio of the distribution range of N projection lines to the area of the image 821 is equal to or greater than a predetermined value.

  FIG. 27 is a diagram showing an example of candidate lines not suitable for calculation of the parameter 46. As shown in FIG. FIG. 27A shows four candidate lines parallel to one another. When four candidate lines are parallel, even if the candidate lines are parallelly moved in the direction of the arrow 2101, the error does not change, and it may be difficult to fix the positions of the candidate lines.

  FIG. 27B shows two candidate lines existing on the same straight line. When two candidate lines exist on the same straight line, the error does not change even if the candidate lines are expanded or contracted in the direction of the arrow 2102, and it may be difficult to fix the scale.

Therefore, in S26 of FIG. 16, the generation unit 28 may select N candidate lines that satisfy the following conditions, and generate N corresponding pairs.
(C21) At least two of the N candidate lines are not parallel.
(C22) Any two candidate lines out of N candidate lines do not exist on the same straight line.

For the same reason, the generation unit 28 may generate N corresponding pairs by selecting N feature lines that satisfy the following conditions.
(C31) At least two feature lines out of N feature lines are not parallel.
(C32) Any two feature lines out of N feature lines do not exist on the same straight line.

  In S29, the error of the corresponding pair straight line may be calculated. FIG. 28 is a diagram for describing calculation of an error in the present embodiment (Example 1). In this case, as shown in FIG. 28, the determination unit 31 calculates the average of the area consisting of N corresponding edge-projection line pairs, and calculates the error between the area of each pair and the average area thereof. It is also good.

  Next, the present embodiment (Example 2) will be described. In the first embodiment, the initial model posture candidate is a pair of the rotation component Ri and the translation component Ti, but in the second embodiment, only the rotation component Ri may be a candidate (other than the translation component Ti, the used data, processing, function Etc. are the same as in Example 1.). In this case, as the translational component Ti, a value is set in advance such that the entire image of the CAD model is captured.

  For example, when the size of the CAD model is 10 cm square, it may be fixed as T = (0, 0, 20) [cm]. In the case of the second embodiment, there is an advantage that the data structure of the CAD image DB 11 may be only the rotation component.

  Next, the present embodiment (Example 3) will be described. In the first and second embodiments, the CAD image DB is created in advance, but in the third embodiment, the CAD image DB 11 is created at the same time when performing the initial model posture estimation process. As a result, although the time taken for one initial model posture estimation process becomes long, the memory capacity can be reduced.

  Next, the present embodiment (Example 4) will be described. The fourth embodiment is the same as the first embodiment in the configuration, processing and the like. In the first embodiment, the selection method of the initial model posture candidate is determined by the number of matching points of the intersection by LLAH, but any other method may be used. If a distinguishable pattern is distributed on the surface of the object, for example, candidates may be determined by matching with local feature amounts described in Non-Patent Document 9.

  In this case, the posture estimation unit 27 may extract ORB feature quantities from two of the CAD image and the captured image, perform matching, and use a candidate with the largest number of matching points as the initial model posture.

  According to the present embodiment, it is possible to reduce the amount of calculation for determining the corresponding pair of the straight line in the captured image used for the pose estimation of the camera and the straight line of the 3D model. That is, the amount of processing can be reduced by reducing the processing object by limiting the orientation of the posture and removing the straight line hidden behind the object from the association processing.

  FIG. 29 is a diagram showing a configuration example of a computer used as the estimation device 21 in the present embodiment. The computer 2200 includes a central processing unit (CPU) 2201, a memory 2202, an input device 2203, an output device 2204, an auxiliary storage device 2205, a medium drive device 2206, and a network connection device 2207. These components are connected to one another by a bus 2208. The imaging device 20 may be connected to the bus 2208.

  The memory 2202 is, for example, a semiconductor memory such as a read only memory (ROM), a random access memory (RAM), or a flash memory, and stores programs and data used for image processing. The memory 2202 can be used as the storage unit 33.

  The CPU 2201 (processor) executes a program using, for example, the memory 2202 to execute the CAD data reading unit 22, the CAD image DB generating unit 23, the candidate line extracting unit 24, the image acquiring unit 25, the feature detecting unit 26, It functions as the posture estimation unit 27. Furthermore, the CPU 2201 functions as a generation unit 28, a position calculation unit 29, an error calculation unit 30, and a determination unit 31.

  The input device 2203 is, for example, a keyboard, a pointing device, etc., and is used to input an instruction or information from an operator or a user. The output device 2204 is, for example, a display device, a printer, a speaker, or the like, and is used to inquire or instruct an operator or a user, and to output a processing result. The processing result may be the N corresponding pairs determined by the determination unit 31. The output device 2204 can be used as the output unit 32 of FIG.

  The auxiliary storage device 2205 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device or the like. The auxiliary storage device 2205 may be a hard disk drive. The computer 2200 can store programs and data in the auxiliary storage device 2205 and load them into the memory 2202 for use. The auxiliary storage device 2205 can be used as the storage unit 33.

  A medium drive 2206 drives a portable recording medium 2209 and accesses the recorded contents. The portable recording medium 2209 is a memory device, a flexible disk, an optical disk, a magneto-optical disk or the like. The portable recording medium 2209 may be a Compact Disk Read Only Memory (CD-ROM), a Digital Versatile Disk (DVD), a Universal Serial Bus (USB) memory, or the like. An operator or a user can store programs and data in this portable recording medium 2209 and load them into the memory 2202 for use.

  Thus, the computer readable recording medium for storing the program and data used for image processing is physical (non-temporary) such as the memory 2202, the auxiliary storage device 2205 or the portable recording medium 2209. It is a recording medium.

  The network connection device 2207 is a communication interface that is connected to a communication network such as a Local Area Network, a Wide Area Network, etc., and performs data conversion involved in communication. The computer 2200 can receive programs and data from an external device via the network connection device 2207, load them into the memory 2202 and use them. The network connection device 2207 can be used as the output unit 32 of FIG.

  Note that the computer 2200 need not include all the components shown in FIG. 29, and some components may be omitted depending on the application or conditions. For example, when the portable recording medium 2209 or the communication network is not used, the medium driving device 2206 or the network connection device 2207 may be omitted.

  According to the present embodiment, in the technique for estimating the position and orientation of the imaging device, the specified initial model orientation is determined when superimposing a three-dimensional line segment of CAD data on an edge of an object in a captured image. As the line segment pairs are associated with each other, the processing time can be reduced. In addition, there is no need to manually adjust the initial model posture, and automatic adjustment is possible, so that usability can be improved. Further, since the initial model attitude is specified, the line segments of the CAD data and the edge lines of the image are associated with each other based on the initial model attitude, so that it is possible to prevent erroneous superimposed display.

  While the disclosed embodiments and their advantages have been described in detail, those skilled in the art can make various changes, additions, and omissions without departing from the scope of the present invention as set forth in the claims. I will.

Further, the following appendices will be disclosed regarding the above embodiment.
(Supplementary Note 1)
For each posture of an object observed from any of a plurality of viewpoints, posture information indicating the posture, a viewpoint image indicating an image of the object observed from the viewpoint corresponding to the posture information, and extraction from the viewpoint image A storage unit for storing a plurality of initial posture candidate information, which is information associating the feature amount of the viewpoint image calculated using the plurality of selected feature points;
An image acquisition unit that acquires a captured image obtained by capturing an object;
The feature amount of the captured image and the feature amounts of the plurality of initial posture candidate information are collated, and the plurality of initial posture candidate information is similar to the posture of the object shown in the captured image based on the collation result A collation unit that extracts initial posture candidate information including posture information of a posture to be selected;
A narrowing unit which narrows down a line segment obtained from the shape information of the object based on the position of the viewpoint at which the object in the posture corresponding to the initial posture candidate information extracted by the collation unit is observed;
An associating unit that associates the narrowed line segment with a feature line detected from the captured image;
An estimation unit configured to estimate the position and the direction of the viewpoint with respect to the object in a three-dimensional space from the image of the object based on the correspondence result;
Estimation device characterized by including (Supplementary Note 2)
The initial posture candidate information is information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object,
The number of the initial posture candidate information corresponds to the number of vertexes of the polyhedron and the center of the surface.
(Supplementary Note 3)
The estimation device further includes
Based on the shape information of the object, the plurality of initial posture candidate information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object is generated The estimation apparatus according to any one of appendices 1 and 2, further comprising:
(Supplementary Note 4)
The feature quantity is obtained by calculating invariants to geometrical transformations with respect to a plurality of intersections around each of the intersections of the viewpoint image or the characteristic lines extracted from the captured image. It is a feature amount,
The matching unit measures the number of intersections at which the feature of the feature of the feature line extracted from the viewpoint image matches the feature of the feature of the feature line extracted from the captured image. The estimation apparatus according to any one of appendices 1 to 3, characterized in that posture information having the largest number of times is estimated as the initial posture information.
(Supplementary Note 5)
The estimation device according to any one of appendices 1 to 4, wherein the viewpoint information is represented by a three-dimensional coordinate system including a rotation component.
(Supplementary Note 6)
On the computer
Acquire a captured image obtained by capturing an object,
For each posture of an object observed from any of a plurality of viewpoints from the storage unit, posture information representing the posture, a viewpoint image indicating an image of the object observed from a viewpoint corresponding to the posture information, and Acquiring a plurality of initial posture candidate information which is information associating the feature amounts of the viewpoint image calculated using the plurality of feature points extracted from the viewpoint image;
The feature amount of the captured image and the feature amounts of the plurality of initial posture candidate information are collated, and the plurality of initial posture candidate information is similar to the posture of the object shown in the captured image based on the collation result Extracting initial posture candidate information including posture information of the posture to be
Narrowing down a line segment obtained from the shape information of the object based on the position of the viewpoint at which the object of the orientation corresponding to the extracted initial orientation candidate information is observed;
Associating the narrowed line segment with a feature line detected from the captured image;
Estimating the position and the direction of the viewpoint with respect to the object in the three-dimensional space from the image of the object based on the correspondence result
An estimation program that causes the process to run.
(Appendix 7)
The initial posture candidate information is information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object,
The number of the initial posture candidate information corresponds to the number of vertexes of the polyhedron and the center of the surface.
(Supplementary Note 8)
The estimation program further causes the computer to:
Based on the shape information of the object, the plurality of initial posture candidate information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object is generated The estimation program according to appendix 6 or 7, characterized in that the processing is performed.
(Appendix 9)
The feature quantity is obtained by calculating invariants to geometrical transformations with respect to a plurality of intersections around each of the intersections of the viewpoint image or the characteristic lines extracted from the captured image. It is a feature amount,
In the extraction of the initial posture candidate information, the number of intersections at which the feature amount of the intersection point of the feature line extracted from the viewpoint image matches the feature amount of the intersection point of the feature line extracted from the captured image is measured. The estimation program according to any one of Appendices 6 to 8, wherein posture information having the largest number of intersection points is estimated as the initial posture information.
(Supplementary Note 10)
The estimation program according to any one of appendices 6 to 9, wherein the viewpoint information is represented by a three-dimensional coordinate system including a rotation component.
(Supplementary Note 11)
The computer is
Acquire a captured image obtained by capturing an object,
For each posture of an object observed from any of a plurality of viewpoints from the storage unit, posture information representing the posture, a viewpoint image indicating an image of the object observed from a viewpoint corresponding to the posture information, and Acquiring a plurality of initial posture candidate information which is information associating the feature amounts of the viewpoint image calculated using the plurality of feature points extracted from the viewpoint image;
The feature amount of the captured image and the feature amounts of the plurality of initial posture candidate information are collated, and the plurality of initial posture candidate information is similar to the posture of the object shown in the captured image based on the collation result Extracting initial posture candidate information including posture information of the posture to be
Narrowing down a line segment obtained from the shape information of the object based on the position of the viewpoint at which the object of the orientation corresponding to the extracted initial orientation candidate information is observed;
Associating the narrowed line segment with a feature line detected from the captured image;
Estimating the position and the direction of the viewpoint with respect to the object in the three-dimensional space from the image of the object based on the correspondence result
Estimation method characterized by
(Supplementary Note 12)
The initial posture candidate information is information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object,
The estimation method according to appendix 11, wherein the number of pieces of initial pose candidate information corresponds to the number of vertexes and face centers of the polyhedron.
(Supplementary Note 13)
The computer further comprises:
Based on the shape information of the object, the plurality of initial posture candidate information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object is generated The estimation method according to appendix 11 or 12, characterized in that:
(Supplementary Note 14)
The feature quantity is obtained by calculating invariants to geometrical transformations with respect to a plurality of intersections around each of the intersections of the viewpoint image or the characteristic lines extracted from the captured image. It is a feature amount,
In the extraction of the initial posture candidate information, the number of intersections at which the feature amount of the intersection point of the feature line extracted from the viewpoint image matches the feature amount of the intersection point of the feature line extracted from the captured image is measured. The estimation method according to any one of appendices 11 to 13, wherein posture information having the largest number of intersection points is estimated as the initial posture information.
(Supplementary Note 15)
The estimation method according to any one of appendices 11 to 14, wherein the viewpoint information is represented by a three-dimensional coordinate system including a rotation component.

DESCRIPTION OF SYMBOLS 1 estimation device 2 storage unit 3 image acquisition unit 4 collation unit 5 narrowing-down unit 6 association unit 7 estimation unit 8 generation unit 9 initial posture candidate information 11 CAD image DB
12 captured image 21 estimation device 22 CAD data reading unit 23 CAD image DB generation unit 24 candidate line extraction unit 25 image acquisition unit 26 feature detection unit 27 posture estimation unit 28 generation unit 29 position calculation unit 30 error calculation unit 31 determination unit 32 output Part 33 Storage part 41 CAD data 43 Feature line 44 Candidate line 45 Correspondence pair 46 Parameter 47 Index 48 Parameter

Claims (7)

On the computer
Acquire a captured image obtained by capturing an object,
For each posture of an object observed from any of a plurality of viewpoints from the storage unit, posture information representing the posture, a viewpoint image indicating an image of the object observed from a viewpoint corresponding to the posture information, and Acquiring a plurality of initial posture candidate information which is information associating the feature amounts of the viewpoint image calculated using the plurality of feature points extracted from the viewpoint image;
The feature amount of the captured image and the feature amounts of the plurality of initial posture candidate information are collated, and the plurality of initial posture candidate information is similar to the posture of the object shown in the captured image based on the collation result Extracting initial posture candidate information including posture information of the posture to be
Narrowing down a line segment obtained from the shape information of the object based on the position of the viewpoint at which the object of the orientation corresponding to the extracted initial orientation candidate information is observed;
Associating the narrowed line segment with a feature line detected from the captured image;
Estimating the position and the direction of the viewpoint with respect to the object in the three-dimensional space from the image of the object based on the correspondence result
An estimation program that causes the process to run. The initial posture candidate information is information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object,
The estimation program according to claim 1, wherein the number of pieces of initial posture candidate information corresponds to the number of vertexes and face centers of the polyhedron. The estimation program further causes the computer to:
Based on the shape information of the object, the plurality of initial posture candidate information on the posture of the object observed from each of the viewpoints when the viewpoints are arranged at the centers of vertices and faces of a polyhedron surrounding the object is generated The estimation program according to claim 1 or 2, wherein the processing is performed. The feature quantity is obtained by calculating invariants to geometrical transformations with respect to a plurality of intersections around each of the intersections of the viewpoint image or the characteristic lines extracted from the captured image. It is a feature amount,
In the extraction of the initial posture candidate information, the number of intersections at which the feature amount of the intersection point of the feature line extracted from the viewpoint image matches the feature amount of the intersection point of the feature line extracted from the captured image is measured. The estimation program according to any one of claims 1 to 3, wherein posture information having the largest number of intersection points is estimated as the initial posture information. The estimation program according to any one of claims 1 to 4, wherein the initial posture candidate information is represented by a three-dimensional coordinate system including a rotation component. For each posture of an object observed from any of a plurality of viewpoints, posture information indicating the posture, a viewpoint image indicating an image of the object observed from the viewpoint corresponding to the posture information, and extraction from the viewpoint image A storage unit for storing a plurality of initial posture candidate information, which is information associating the feature amount of the viewpoint image calculated using the plurality of selected feature points;
An image acquisition unit that acquires a captured image obtained by capturing an object;
The feature amount of the captured image and the feature amounts of the plurality of initial posture candidate information are collated, and the plurality of initial posture candidate information is similar to the posture of the object shown in the captured image based on the collation result A collation unit that extracts initial posture candidate information including posture information of a posture to be selected;
A narrowing unit which narrows down a line segment obtained from the shape information of the object based on the position of the viewpoint at which the object in the posture corresponding to the initial posture candidate information extracted by the collation unit is observed;
An associating unit that associates the narrowed line segment with a feature line detected from the captured image;
An estimation unit configured to estimate the position and the direction of the viewpoint with respect to the object in a three-dimensional space from the image of the object based on the correspondence result;
An estimation apparatus comprising: The computer is
Acquire a captured image obtained by capturing an object,
For each posture of an object observed from any of a plurality of viewpoints from the storage unit, posture information representing the posture, a viewpoint image indicating an image of the object observed from a viewpoint corresponding to the posture information, and Acquiring a plurality of initial posture candidate information which is information associating the feature amounts of the viewpoint image calculated using the plurality of feature points extracted from the viewpoint image;
The feature amount of the captured image and the feature amounts of the plurality of initial posture candidate information are collated, and the plurality of initial posture candidate information is similar to the posture of the object shown in the captured image based on the collation result Extracting initial posture candidate information including posture information of the posture to be
Narrowing down a line segment obtained from the shape information of the object based on the position of the viewpoint at which the object of the orientation corresponding to the extracted initial orientation candidate information is observed;
Associating the narrowed line segment with a feature line detected from the captured image;
Estimating the position and the direction of the viewpoint with respect to the object in the three-dimensional space from the image of the object based on the correspondence result
Estimation method characterized by