JP5988368B2 - Image processing apparatus and method - Google Patents

Description

  The present invention relates to an image processing apparatus and method for estimating a posture of a reference coordinate system using a single index and a geometric feature.

  AR (augmented reality) is a concept of adding virtual information such as characters and CG to the real world. Among them, in order to superimpose a three-dimensional CG on a captured image, imaging with respect to a reference coordinate system in the real world is performed. It is necessary to estimate the attitude of the apparatus (or the attitude of the reference coordinate system with respect to the imaging apparatus as equivalent information representing the opposite).

  As an application of the posture estimation technique of the reference coordinate system, a method for measuring the size of a three-dimensional object captured in a captured image using the estimated reference coordinate system has been proposed. Is disclosed.

  Patent Document 1 discloses a method of measuring a three-dimensional dimension by estimating the three-dimensional vertex coordinates of a rectangular parallelepiped object, and it is possible to measure with high accuracy by using all the vertices of a rectangular parallelepiped object as a constraint condition. Is possible.

  In Patent Document 2, a plane projection transformation matrix is calculated from an index affixed to the upper surface of a rectangular parallelepiped object, and the dimensions on each surface of the object are measured by applying rotation and translation of the reference coordinate system to the plane projection transformation matrix. In other words, an object such as a cone whose angular relationship between the bottom surface and the side surface is known in advance can be measured.

JP 2008-122109 A Japanese Patent Laid-Open No. 2008-140047

  In Patent Document 1, by limiting the measurement object, all the geometric shapes of the measurement object can be used as constraints, so that the reference coordinate system can be estimated with high accuracy, while the shape of the measurable object is a rectangular parallelepiped. There is a problem that it is limited to objects.

  In Patent Document 2, only a single index vertex or edge constraint condition can be used for calculation of the planar projective transformation matrix, and the estimation accuracy of the reference coordinate system is not stable compared to the method of Patent Document 1 There is. In particular, an estimation error is likely to occur in the vertical direction compared to the horizontal direction of the plane where the index exists, in that only the constraint condition on a specific surface is used.

  The present invention has been made in view of the above-described problems, and a first object thereof is to estimate a reference coordinate system with high accuracy under a limited constraint condition.

  A second object of the present invention is to estimate the size with high accuracy without necessarily limiting the target to a specific shape such as a rectangular parallelepiped, based on the arrangement estimated with high accuracy.

  In order to achieve the above object, according to the present invention, a reference coordinate system defined by an index on a plane on which a predetermined index is set is obtained based on a captured image captured by an imaging apparatus, the index, a parallel relationship on the plane, and / or Or an image processing device that uses a predetermined geometric feature configured as an orthogonal relationship, and an index detection unit that detects the index from the captured image, and the index is expressed in the reference coordinate system Based on a predetermined initial arrangement and an arrangement of the detected index on the captured image, an initial attitude of the reference coordinate system with respect to the imaging apparatus is calculated, and a conversion relationship represented by the initial attitude is large. In the detected geometric feature, an initial posture calculating unit that decomposes into each element corresponding to the position, orientation, and distortion, a geometric feature detecting unit that detects the geometric feature from the captured image, and An initial arrangement estimating unit for estimating an initial arrangement of the geometric feature in the reference coordinate system based on the elements of the resolution, the size, the position, and the orientation, and a predetermined initial arrangement of the index, and A reference for estimating a posture of the reference coordinate system with respect to the imaging device based on an overall correspondence relationship between the estimated initial arrangement of the geometric feature and the arrangement of the detected index and the geometric feature on the captured image. A coordinate system estimation unit is provided as a first feature.

  In addition, the present invention further includes a dimension measuring unit that measures a dimension of a predetermined location on the captured image based on the estimated orientation of the reference coordinate system and a known size of the predetermined index. The second feature.

  According to the first feature, it is possible to estimate the initial arrangement of geometric features in the reference coordinate system and estimate the attitude of the reference coordinate system with respect to the imaging apparatus with high accuracy by using the initial arrangement of the indices in the reference coordinate system. In this case, in addition to the use of known indices, geometric features that are configured only as parallel relations and / or orthogonal relations and have no other constraint conditions such as relative arrangement with the indices are used. Compared with the case where a single index is originally used, it is possible to perform estimation with higher accuracy. Therefore, the first object is achieved.

  According to the second feature, since the dimension measurement is performed using the reference coordinate system estimated with high accuracy, the dimension measurement with high accuracy becomes possible. At this time, the target is limited to a specific shape such as a rectangular parallelepiped. Since it is not necessary, the second objective is achieved.

1 is a configuration diagram as an embodiment of an AR system using an image processing apparatus of the present invention. It is a figure which shows the rectangular parallelepiped as an example of an imaging target. It is a functional block diagram of an image processing apparatus. It is a figure for demonstrating (1) the square parameter | index as an example of a parameter | index, (2) the reference coordinate system defined by a parameter | index, and (3) providing the frame area for a detection in a parameter | index. It is a figure for demonstrating the parallel line as an example of a geometric feature. It is a figure for demonstrating the orthogonal line as an example of a geometric feature. It is a figure for demonstrating typically that a reference coordinate system and a picked-up image system are mutually converted by a plane projection transformation matrix and its inverse matrix. It is a figure for demonstrating the process for removing the said distortion when forming a parallelogram in a reference coordinate system, although it is distorted in a captured image. FIG. 8 is a diagram for explaining a process for further removing the distortion when the parallelogram forms a square in the reference coordinate system after the distortion removal process described in FIG. 7; It is a figure for demonstrating the summary of the process in an initial arrangement | positioning estimation part. It is a series of figures for demonstrating the meaning of a geometric feature. It is a series of figures for demonstrating the meaning of a geometric feature. It is a series of figures for demonstrating the meaning of a geometric feature. It is a series of figures for demonstrating the meaning of a geometric feature. It is a series of figures for demonstrating the meaning of a geometric feature. It is a figure for demonstrating that it can detect from the various geometric features which are not restricted to one geometric feature, and selection when a several parallel relationship and orthogonal relationship are detected. It is a figure for demonstrating the geometric feature comprised by a cylinder as an example using the geometric feature on another plane. It is a figure for demonstrating the geometric feature comprised by a cone as an example using the geometric feature on another plane. It is a functional block diagram of an image processing device in an embodiment using a plurality of indices. It is a figure for demonstrating the significance of embodiment which uses two or more indicators.

  FIG. 1 shows a configuration diagram as an embodiment of an AR system using an image processing apparatus according to the present invention, and the outline of the present invention will be described with reference to FIG. The AR system 1 includes an imaging device 10, an information display device 11, a database 12, and an image processing device 2. The image processing apparatus 2 contributes to the AR system 1 by estimating the reference coordinate system using the index and the arrangement information of the plurality of geometric features, and measuring the dimensions based on the estimation result.

  The imaging device 10 is a device that acquires a camera image. In addition to a commercially available WEB camera, a camera module mounted on a mobile phone terminal may be used. In the imaging apparatus 10 of the present invention, it is assumed that internal parameters such as a focal length, an optical axis shift, and a distortion parameter are known by prior calibration.

  The imaging target is a single index in a general AR system, while the present invention includes a plurality of geometric features in addition to the single index. At this time, the index and all the geometric features are not necessarily on the same plane. In FIG. 1, an example of such an imaging target in the real world is shown in (1). That is, the index 1 and the geometric feature 2 exist on the same plane P1, and separately from this, the geometric feature 3 exists on another plane P2.

  Hereinafter, the case where the index and the geometric feature are on the same plane as P1 will be described first, and then the case where the geometric feature is also on another plane such as P2. Note that (2) and (3) in FIG. 1 will be referred to in later description.

  In addition, geometric features can be used in accordance with indices, and are configured to detect two sets of parallel lines and / or orthogonal lines on a plane as schematically shown in FIG. It is a thing. Details will be described later.

  According to the image processing apparatus 2, the index and the geometric feature are detected, and in the example of FIG. 1, the reference coordinate system XYZ on the plane P1 defined by the index 1 as shown in (2) is set as the attitude with respect to the imaging apparatus 10. Can be estimated. Furthermore, based on the estimation result, the image processing apparatus 2 can measure the size of the imaging target.

  FIG. 2 shows a rectangular parallelepiped as an example of the imaging target, and shows that the sides S1 to S9 are visible on the captured image. The image processing apparatus 2 estimates the reference coordinate system on the upper surfaces S1-S8-S2-S7 of the rectangular parallelepiped where the index 1 is installed by detecting the index 1 and the geometric feature, and the upper surface S1-S8-S2 -Measure dimensions of each side of S7 and user-specified predetermined locations on the surface.

  Furthermore, in the case of using geometric features on a different plane from the index 1 described later, the dimensions S1 to S9 corresponding to the width, depth, or height of the rectangular parallelepiped and the user-specified predetermined location on each surface are measured. It can be performed with high accuracy. At this time, the size only needs to be known only for the index 1, and for the rectangular parallelepiped, it is only necessary to know that the solid is a rectangular parallelepiped, and the dimensions of each unknown side can be measured.

  As the geometric feature, a rectangle formed by the upper surfaces S1-S8-S2-S7 on the same plane as the index 1 can be used. Furthermore, in the case of using a geometric feature on a different plane from the index 1 described later, the orthogonal relationship and / or the parallel relationship between the sides regarding part or all of the sides S1 to S9 derived from the target being a rectangular parallelepiped. Can be used. Specifically, as shown in the figure, the sides belonging to each set of sides S1 to S3, sides S4 to S6, and sides S7 to S9 are parallel to each other, and a geometric feature based on the relationship can be used.

  The information display device 11 is a device for displaying text, moving images, CG (computer graphics) or the like, and may use a PC (personal computer), a television display, or a head-mounted display. In addition, the imaging device and the information processing device may share a casing like a mobile phone terminal. In addition, it is desirable that the touch panel is mounted so that the user can specify a line segment to be measured by pointing at the time of dimension measurement described later.

  In the AR system, it is common to superimpose and display CG or the like on the moving image acquired by the imaging apparatus. In this embodiment, CG or the like may be displayed at an arbitrary position in the estimated reference coordinate system. . Further, if dimension measurement is performed, only measured dimension information may be displayed.

  The database 12 is a device for recording data, and may be a non-volatile memory storage such as a hard disk or a flash memory, or a volatile memory generally used for a main storage device of a PC. In the present embodiment, at least the internal parameters of the imaging device 10 and the shape and size of the index are recorded and used for the image processing device 2. However, the image processing device 2 provides necessary information according to the usage environment. You may keep it.

  The database 12 may further record a CG model to be superimposed and other accompanying information. As an example of the applied use of the present invention, the additional information records information additionally displayed when the dimensions are measured, for example, a delivery fee for each dimension when the dimensions of a cardboard box for delivery are measured. May be.

  FIG. 3 is a functional block diagram of the image processing apparatus 2. The image processing apparatus 2 includes a captured image acquisition unit 3, an index detection unit 4, an initial posture calculation unit 5, a geometric feature detection unit 6, an initial arrangement estimation unit 7, a reference coordinate system estimation unit 8, and a dimension measurement unit 9. The initial posture calculation unit 5 includes a matrix calculation unit 51 and a matrix decomposition unit 52. The initial arrangement estimation unit 7 includes a distortion correction unit 71, a matrix configuration unit 72, and an estimation unit 73. The outline of each part is as follows.

Note that in the present invention, the following distinctions are used with respect to the terms for the objects handled by the respective parts, particularly the terms referring to the relationship in space or plane.
(1) “Attitude”: Position and orientation of an index (index coordinate system or reference coordinate system) with respect to an imaging apparatus (imaging apparatus coordinate system) expressed as a planar projective transformation relationship, or vice versa, Term indicating position and orientation. It is a 6-dimensional vector parameter that determines the position and orientation in 3D space, and is generally expressed as “position and orientation” because it refers to the position and orientation. To do.
(2) “Position and orientation”: a term indicating the position and orientation of an index or geometric feature in one plane of a reference coordinate system obtained by applying a plane projection transformation matrix to an index of a pixel coordinate system.
(3) “Arrangement”: a term indicating the positional relationship between indices in one plane of the reference coordinate system or in the pixel coordinate system. In this article, “position and orientation” in (2) is mainly used when referring to a single object (indicator or geometric feature), whereas “placement” is mainly used when referring to multiple objects. Used to indicate their relative positional relationship.

  The captured image acquisition unit 3 acquires a captured image from the imaging device 10 and inputs the acquired image to the index detection unit 4 and the geometric feature detection unit 6. The index detection unit 4 detects an index from the input captured image and inputs it to the initial posture calculation unit 5. The geometric feature detection unit 6 detects a geometric feature from the input captured image and inputs it to the initial arrangement estimation unit 7.

  First, in the matrix calculation unit 51, the initial attitude calculation unit 5 calculates the initial attitude of the index with respect to the imaging device 10 from the index detected by the index detection unit 4. That is, the initial posture is also a posture with respect to the imaging device 10 in the reference coordinate system represented by the index. At the time of the calculation, the initial arrangement as the arrangement of the index in the reference coordinate system, which is recorded in the database 12 or held in the image processing apparatus 2 in advance, and the index detected by the index detection unit 4 An initial posture is calculated by obtaining a planar projective transformation between the arrangement on the captured image.

  The initial posture is referred to as an “initial” posture as so-called intermediate data in the previous stage because the subsequent reference coordinate system estimation unit 8 obtains the posture of the reference coordinate system with higher accuracy. The initial arrangement represents the arrangement of indices and geometric features in the reference coordinate system. The initial arrangement of indices is known in advance, while the initial arrangement of geometric features is unknown. Since it is estimated based on the above-mentioned initial posture, it will be referred to as an initial arrangement.

  The initial posture calculation unit 5 further decomposes the planar projective transformation represented by the initial posture in the matrix decomposition unit 52 to obtain predetermined information necessary for the initial layout estimation process of the geometric feature in the initial layout estimation unit 7. The information is input to the initial arrangement estimation unit 7.

  The initial arrangement estimation unit 7 estimates the initial arrangement in the reference coordinate system of the geometric features detected by the geometric feature detection unit 6 based on the information for determining the initial arrangement of the geometric features input from the matrix decomposition unit 52, Input to the reference coordinate system estimation unit 8. Since the initial arrangement is estimated based on the information from the matrix decomposition unit 52 based on the initial posture of the initial posture calculation unit 5, it is referred to as such.

  The reference coordinate system estimation unit 8 is configured to input the initial arrangement of geometric features and the initial arrangement of the same indices used by the matrix calculation unit 51 to calculate the initial posture, and on the captured image of the geometric features and indices. The orientation of the reference coordinate system determined by the index with respect to the imaging device 10 is estimated from the overall correspondence relationship with the arrangement in FIG. The estimation result becomes one output of the image processing apparatus 2 and is also input to the dimension measuring unit 9.

  The dimension measuring unit 9 measures another dimension of a predetermined portion of the imaging target based on the input orientation of the reference coordinate system and the size of a known index, thereby providing another output of the image processing apparatus 2.

  The details of the processing of each unit will be described below.

  The captured image acquisition unit 3 acquires a captured image from the imaging device 10 and inputs the acquired image to the index detection unit 4 and the geometric feature detection unit 6. The image size of the captured image acquired by the captured image acquisition unit 3 from the imaging device 10 is not limited, but a higher resolution is desirable from the viewpoint of coordinate rounding errors in the index and geometric feature detection processing described later, and the imaging device 10 It is desirable to pass the captured image directly to the index detection unit 4 and the geometric feature detection unit 6. When the load is large due to real-time processing or the like, it may be delivered after being lowered to a predetermined resolution.

  The index detection unit 4 detects the index from the input captured image and inputs the detected index to the initial posture calculation unit 5. An object that can be used as an index has at least four points whose sizes and arrangements are known (this state is expressed as a known shape), and can identify a corresponding point in the captured image for each. I just need it. Here, in order to enable posture calculation by the matrix calculation unit 51 in the subsequent stage, it is necessary to have four or more points. Further, it may be possible to obtain the correspondence of a line segment instead of a point, or an arbitrary index that can be used in the same manner as the four or more points or line segments and that can be identified in a captured image may be used.

  (1) in FIG. 4 shows a general square index in the AR system, and by recognizing the internal pattern, the corresponding points of vertex A, vertex B, vertex C, and vertex D are taken from the captured image. Can be extracted. The arrangement of the points or line segments in the index can be specified by a reference coordinate system set to an arbitrary position and orientation on the index. In the present invention, the marker placement information in the reference coordinate system in such a setting is determined in advance as the initial placement.

  For example, when the index is a square index, the center of gravity of the square index may be set as the origin of the reference coordinate system as shown in (2) of FIG. As a result, the coordinates of the four vertices are determined, and the coordinates of the vertices in a known reference coordinate system and the corresponding coordinates of the vertices in the captured image are set as an initial arrangement, and at least four or more vertex coordinates are calculated as initial postures. Input to part 5.

  In order to simplify the index detection process, a frame area may be provided in the index as shown in (3) of FIG. Thus, the index region can be reliably extracted by binarizing the captured image, and the direction of the index can be detected by recognizing the inside of the index region. When it is difficult to provide a frame, the index region may be detected by direct designation from the user, such as pointing using a display equipped with a touch panel. The various processes and devices for detecting the index described above can be similarly applied to the detection of geometric features described below.

  The geometric feature detection unit 6 detects a geometric feature from the input captured image and inputs it to the initial arrangement estimation unit 7. The geometric feature does not need to be completely known in advance like an index, and can be detected with a higher degree of freedom than various geometric shapes existing in the real world. That is, the geometric feature does not need to have a point whose arrangement is known like an index, and any geometric feature may be used as long as a specific geometric feature existing in the geometric shape can be specified in the captured image.

  Here, the geometric feature specifically refers to a parallel line or an orthogonal line. 5A and 5B show an example of a geometric shape having a corresponding geometric feature. The geometric shape shown in FIG. 5A has only a constraint condition that the line segment AB and the line segment DC and the line segment AD and the line segment BC are parallel to each other, and there is a constraint condition regarding the length of the side. Not. Similarly, the right triangle shown in FIG. 5B has only a constraint that the line segment CA and the line segment CB are at right angles.

  That is, in the example of the index shown in FIG. 3, the coordinates of the vertex A, vertex B, vertex C, and vertex D in the reference coordinate system are known, and the position and orientation in the square reference coordinate system composed of the four points are known. On the other hand, the position and orientation in the reference coordinate system are unknown for the geometric shapes shown in FIGS. 5A and 5B.

  The geometric feature detection unit 6 extracts the geometric features as such parallel relationship and / or orthogonal relationship from the captured image. At this time, in the same manner as the index detection unit 4, the line segment (excluding the line segment constituting the index) is detected from the captured image by edge detection, and the parallel relationship and / or the orthogonal relationship is specified for the predetermined line segment. You just have to do it. At this time, the parallel relationship and / or the orthogonal relationship may be automatically determined by a known color feature in the vicinity of the line segment.

  For example, if a red rectangular signboard of an arbitrary size is installed on a white wall and the parallelism and right angle in the rectangle are predetermined to be used as geometric features, edge detection and color features around the edges The boundary of the rectangle may be automatically detected by confirmation, and the parallel relationship and / or the orthogonal relationship may be obtained from each side on the boundary.

  Alternatively, the user is allowed to visually check the detected line segment, and the user himself / herself determines which line segment represents the parallel relationship and / or the orthogonal relationship, and the information on the determination is described above. The geometric feature detection unit 6 may receive the information via a pointing operation or the like. For example, in the case of FIG. 2, the user may determine from the detected line segments S1 to S9.

  Although the description has been given using the line segment, the point corresponding to both ends of the line segment by the point detection as in the index detection unit 4, or both the line segment and the point, or the equivalent parallel relationship and / or Alternatively, the geometric feature detection unit 6 can perform the same processing for any object that can specify an orthogonal relationship.

  In the initial posture calculation unit 5, first, the matrix calculation unit 51 obtains the initial posture of the reference coordinate system as a planar projective transformation matrix from the index detected on the captured image. Thereafter, the matrix decomposition unit 52 extracts information used when determining the initial posture of each geometric feature in the matrix configuration unit 72 described later, and inputs the information to the matrix configuration unit 72.

  Specifically, the information to be used is a similarity transformation matrix obtained by decomposing a planar projection transformation matrix corresponding to the initial posture of the index into a product of a similarity transformation matrix, an affine transformation matrix, and a projection transformation matrix, or a similarity It is a transformation matrix and an affine transformation matrix. The former (similar transformation matrix) corresponds to the first embodiment described later, and the latter (similar transformation matrix and affine transformation matrix) corresponds to the second embodiment.

  The initial posture of the index in the matrix calculation unit 51 is calculated by calculating a plane projection transformation matrix from the correspondence of at least four sets of points on the same plane or the correspondence of line segments. Since this calculation is common in an AR system that superimposes and displays CG, details are omitted. In the following, the planar projective transformation matrix is described as a mapping from the reference coordinate system to the captured image as conceptually shown in FIG. 6, but it is only a matter of definition which is called forward mapping or reverse mapping. Since there is not, you may obtain | require as this reverse mapping.

In FIG. 6, as an example of the index, a square index as shown in FIG. 4 is converted from the reference coordinate system to the captured image system by the plane projection transformation matrix H according to the definition of the matrix, and again by the inverse matrix H− ^1. It has been converted to the reference coordinate system. That is, when the point in the reference coordinate system is (X _m , Y _m ), the corresponding point in the captured image system is (u, v), and s is the scale factor, the plane projection transformation matrix H that the matrix calculation unit 51 obtains is 3 × 3 matrix,

  The following relational expression (Formula 1) holds.

A mathematical relationship that is a premise in the processing of the matrix decomposition unit 52 will be described. According to Non-Patent Document 1, an arbitrary plane projective transformation matrix H _any includes a similarity transformation matrix H _S corresponding to the transformation of size, position, and orientation, an affine transformation matrix H _A corresponding to transformation of affine distortion, and projection distortion. and the projective transformation matrix H _P corresponding to the conversion,

(Equation 2) to (Equation 4) of (Equation 3) under the constraint that det (K) = 1 for K in (Equation 3)
H _any = H _S H _A H _P (Formula 5)
Can be uniquely decomposed into the form The matrix decomposition unit 52 performs the above decomposition on the inverse matrix H ⁻¹ of the matrix H calculated by the matrix calculation unit 51 as shown in (Equation 6). In the first embodiment described later, the similarity conversion matrix H _S In the second embodiment, the similarity transformation matrix H _S and the affine transformation matrix H _A are input to the matrix configuration unit 72. The reason why the decomposition is performed on the inverse matrix H ⁻¹ is to make it suitable for processing in the matrix construction unit 72 described later.
H ⁻¹ = H _S H _A H _P (Formula 6)

  [Non-Patent Document 1] R. Hartley and A. Zisserman: "Multiple view geometry in computer vision, second edition," Cambridge University Press, 2003.

  The initial arrangement estimation unit 7 estimates the initial arrangement of the geometric features in the reference coordinate system based on the information for determining the initial posture of the geometric features input from the matrix decomposition unit 52. Specifically, first, in the case of the second embodiment, the distortion correction unit 71 calculates a projection transformation matrix that removes the projection distortion from the captured image based on the geometric feature, and in the case of the first embodiment. In addition to the projective transformation matrix, an affine transformation matrix for further removing affine distortion from the captured image from which the projection distortion has been removed is calculated.

  The matrix construction unit 72 constructs a planar projective transformation matrix as the product of the transformation matrix calculated by the distortion correction unit 71 and the matrix input from the matrix decomposition unit 52. Further, the estimation unit 73 estimates the initial arrangement of the geometric features using the configured planar projective transformation matrix and inputs it to the reference coordinate system estimation unit 8.

  Hereinafter, after describing the procedure for calculating the corresponding transformation matrix from the geometric feature in the distortion correction unit 71, the procedure for estimating the initial posture of the geometric feature by constructing the planar projection transformation matrix in the matrix construction unit 72 and the estimation unit 73 State.

According to Non-Patent Document 1, the line at infinity be specified direction in the same plane of the reference coordinate system from two different sets of parallel lines, it is possible to obtain the H _P corresponding to the projection distortion. The present invention utilizes the H _P distortion removal in geometric features.

That is, when the homogeneous coordinate representation of the infinity line in the captured image is l = (l ₁ , l ₂ , l ₃ ), H ₁ defined by the following (Equation 7) is (Equation 5) or ( Corresponds to _HP in equation 6).

From H _P = H ₁ , if the point of the captured image is x,
x '= H _P x (Equation 8)
Thus, the projection distortion can be removed from the photographed image, so that the present invention applies this relationship to the geometric feature.

For example, when a quadrangle abcd that appears distorted on the captured image as shown in FIG. 7 forms a parallelogram in the reference coordinate system, that is, a set of line segments that form parallel lines in the reference coordinate system is a line segment ab And a line segment dc and a line segment ad and a line segment bc, the point a, the point b, the point c, and the point d are set as homogeneous coordinates,
l = ((a x b) x (d x c)) x ((a x d) x (b x c)) (Equation 9)
(× is an outer product) and with the above configuration, it is possible to calculate H _P = H ₁ in (Equation 7), so the distortion of geometric features that originally have a parallel relationship is eliminated. Thus, it is possible to return to a state having a parallel relationship. As shown in FIG. 7, the infinity straight line l is configured as a straight line passing through the intersection point e between the straight lines ad and bc and the intersection point f between the straight lines ab and cd.

Further, according Similarly, non-patent document 1, the two pairs of orthogonal lines in the same plane of the reference coordinate system, is possible to obtain the H _A further remove affine distortion from the captured image projection distortion has been removed by the H _P it can. In the present invention, this relationship can also be applied to geometric features.

  That is, a set of line segments that intersect perpendicularly in the reference coordinate system is defined by line segment l and line segment m, line segment j and k, and each is expressed in homogeneous coordinates in a captured image from which projection distortion has been removed. And At this time,

For H ₂ K defined as

And under s ₁₂ = s ₂₁ and s = (s ₁₁ , s ₁₂ , s ₂₂ ) ^T ,

The following relational expression (Formula 12) can be constructed. In the above equation, if the orthogonal lines are not parallel, that is, if the line segment l and the line segment j are not parallel, the vector s is obtained, and K is obtained by decomposing the matrix S determined by (Equation 11) using the elements of the vector s. It is possible to obtain H ₂ from (Equation 12). Here, the inverse matrix H ₂ ^-1 of the H ₂ and corresponds to H _A in (Equation 5) or (6), (Equation 8) x 'points of captured images perspective distortion has been removed by Then
x '' = H _A x '... (Formula 13)
Thus, affine distortion is further removed from the captured image from which the projection distortion has been removed.

For example, as a state after the distortion removal processing described with reference to FIG. 7, when a quadrangle (parallelogram) abcd as shown in FIG. 8 forms a square in the reference coordinate system, that is, an orthogonal line is formed in the reference coordinate system. When a set of line segments is defined by a line segment ab and a line segment ad and a line segment ac and a line segment bd, the point a, the point b, the point c, and the point d are represented as homogeneous coordinate expressions.
l = a × b (Formula 14)
m = a × d (Equation 15)
j = a × c (Equation 16)
k = b × d (Equation 17)
From these four line segments, K can be calculated by the above-described (Equation 11) or the like, and accordingly, H ₂ = _HA can be calculated further from (Equation 10).

  As is clear from the processing details of the distortion correction unit 71 described above, the geometric feature is configured such that at least two sets of parallel lines and at least two sets of orthogonal lines exist on the same plane (first embodiment). Can obtain a projective transformation matrix and an affine transformation matrix. Further, when the geometric feature is configured such that at least two sets of parallel lines exist on the same plane (referred to as the second embodiment), a projective transformation matrix can be obtained.

  As an example of the geometric feature, for example, a rectangle is a special case of a parallelogram having two sets of parallel lines, so the second embodiment can be applied. However, if one point is selected from the four vertices of the rectangle and the existence of a pair of orthogonal lines intersecting at the point is specified, the orthogonality intersecting at the remaining three vertices from the two specified parallel line conditions The line is automatically determined. Therefore, since the rectangle is a geometric feature having two sets of parallel lines and one set of orthogonal lines, the first embodiment cannot be applied.

  On the other hand, if the rectangle is further limited to a square, then two sets of parallel lines are specified, and then one set of orthogonal lines intersecting at one of the four vertices and one set of orthogonal lines formed by diagonal lines. Lines can be defined. Therefore, the first embodiment can be applied to a square.

  FIG. 9 conceptually shows the relationship of conversion by the distortion correction unit 71 including the relationship of processing by the matrix construction unit 72 and the estimation unit 73 described below.

That is, as a typical example, when the geometric features have two sets of parallel relationship and orthogonal relationship, such as a square, even if the captured image is distorted as shown in (1), (Equation 7) to (Equation) 9) seeking elements H _P of the perspective distortion as described in the distortion is removed as shown in (4) can be formed as parallelogram state as shown in (2). Further, as described in (Expression 10) to (Expression 17), the element _{HA of the} affine distortion is obtained and the distortion is removed as shown in (5), and the square state as shown in (3) is obtained. Can be made.

In (3), the geometric features from which the distortion has been removed by the matrix construction unit 72 and the estimation unit 73 are further placed in the reference coordinate system of the index. That is, as indicated by Hs in parentheses in (5), the matrix component 72 further corresponds to the similarity transformation matrix H _s in (Equation 5) or (Equation 6) from the matrix decomposition unit 52. By receiving and constructing the matrix H, the initial arrangement of the geometric features in the reference coordinate system can be obtained by the estimation unit 73 as shown in (3). With the matrix H, the geometric features can be mutually converted between the reference coordinate system and the captured image system as shown in (7) and (6).

  FIG. 9 shows the case of the first embodiment in which the geometric features have both a parallel relationship and an orthogonal relationship. However, even in the case of only the parallel relationship in the second embodiment, (5 By omitting the part), mutual conversion can be performed in the same manner.

Subsequently, a procedure for constructing a planar projective transformation matrix (H ′ ⁻¹ described below) with geometric features as constraint conditions by the matrix construction unit 72 and estimating the initial arrangement of the geometric features in the reference coordinate system by the estimation unit 73 explain.

When the projection transformation matrix H ′ _P and the affine transformation matrix H ′ _A are obtained from the geometric feature according to the first embodiment, the inverse matrix H ⁻¹ of the planar projection transformation matrix H obtained from the index is decomposed by the matrix decomposition unit 52. The similarity transformation matrix H _S is input to the matrix construction unit 72. The inverse matrix H ⁻¹ of the planar projection transformation matrix of the index is
H ⁻¹ = H _S H _A H _P (Equation 18)
The matrix construction unit 72 obtains an inverse matrix H ′ ⁻¹ of the new planar projection geometric matrix H ′ for the geometric feature,
H ′ ⁻¹ = H _S H ′ _A H ′ _P (Equation 19)
Can be obtained as follows.

Then, the estimation unit 73 obtains an initial arrangement of geometric features by converting all points in the captured image used to obtain H ′ _A and H ′ _P to points in the reference coordinate system by H′− ^1. . Note that the points used to determine H ′ _A and H ′ _P may be the start point and end point of the line segment used to determine these, or two arbitrary points on the line segment. You may convert in the state of a line segment.

Similarly, when only the projection transformation matrix H ′ _P is obtained from the geometric feature according to the second embodiment, the similarity transformation matrix H _S obtained by decomposing the inverse matrix of the planar projection transformation matrix H obtained from the index by the matrix decomposition unit 52 and The affine transformation matrix H _A is input to the matrix configuration unit 72. At this time, the inverse matrix H′− ¹ of the new planar projective geometric matrix H ′ for the geometric feature is
H ' ^-1 = H _S H _A H' _P (Equation 20)
The initial arrangement of the geometric features in the captured image in the reference coordinate system can be calculated in the same manner as described above in the first embodiment. With the above processing, the initial position and orientation of each geometric feature is calculated.

  The reference coordinate system estimation unit 8 uses the initial arrangement of indices given in advance in the reference coordinate system and the initial arrangement of geometric features obtained in the reference coordinate system to determine the attitude of the reference coordinate system with respect to the imaging device 10. presume. About this estimation calculation itself, it becomes a process equivalent to the matrix calculation part 51. FIG. That is, since the initial arrangement of the index and the geometric feature is already known in the reference coordinate system estimation unit 8, each point of the corresponding index and the geometric feature on the captured image with the initial arrangement of the index and the geometric feature as a constraint condition Alternatively, the posture of the imaging device 10 in the reference coordinate system, that is, the posture of the reference coordinate system with respect to the imaging device 10 is estimated from the overall correspondence with each line segment.

  At this time, the initial value is calculated from the position and orientation parameters of the imaging device obtained from the planar projection transformation matrix obtained based on the correspondence of the points or line segments, and the reprojection error function is minimized using an iterative method. The position and orientation parameters may be optimized.

  The dimension measuring unit 9 can measure the dimensions of a three-dimensional object or the like using the estimated reference coordinate system. At this time, information for specifying a location and a plane to be measured is required, but it is also possible to use a user instruction such as pointing. Represent the measurement location in the reference coordinate system, find its size (given by the length as a value on the coordinate system), find the ratio of the size to the index size in the reference coordinate system, By multiplying the actual dimension, the actual dimension of the measurement location is measured. In addition, when a measurement location differs from the plane in which the parameter | index and the geometric feature were installed, it mentions later.

10A to 10E are a series of diagrams for explaining the significance of geometric features. As shown in FIG.10A, when using an index (referred to as index 1), it corresponds to the orientation of the imaging device 10 between the captured image system shown in (1) and the reference coordinate system defined by index 1 shown in (2). The plane projective transformation matrix H ₁ to be obtained can be obtained as shown in (3). In the present invention, the matrix calculation unit 51 takes charge of this.

Here, the matrix H ₁ or the like is used in the reverse form of the definition form of FIG. H ₁ and H ₂ are finally obtained by the estimation unit 73 for index 1 and geometric feature 2 here, unlike those of (Equation 7) and (Equation 10), which represent intermediate processing in the matrix construction unit 72. Represents a planar projective transformation matrix.

On the other hand, if the plane projection transformation matrix H ₁ is applied to the geometric feature 2 on the same plane as shown in (4), the shape is distorted due to an error until the calculation of the matrix H ₁ , so that (2) As shown, the correct arrangement in the reference coordinate system cannot be obtained. In the present invention, an initial arrangement is obtained on the assumption that an approximate arrangement according to the correct arrangement of the geometric feature 2 is given.

As shown in (4) and (5) of FIG. 10B, if the geometric feature 2 has the known original shape like the index 1, the planar projective transformation matrix H ₂ is the same as the index 1. However, in this case, shape information is required in the same way as the index, and an arrangement relationship with the index 1 is also necessary for highly accurate posture estimation, and even if the arrangement relationship is actually satisfied exactly There is a disadvantage in that errors cannot be avoided.

  Note that, as described in Patent Document 1, generally, the more the constraint conditions are, the higher the estimation of the reference coordinate system becomes possible. Until now, only complete constraint conditions that uniquely determine the shape of the object (coordinates of all vertices are known in the case of a rectangular parallelepiped) can be used. For example, the object may be partially parallel or orthogonal. Even if we had the information that we had, there was no way to make effective use of it. In the present invention, the limited constraint condition is utilized to the maximum extent with respect to the object whose constraint condition has been relaxed, thereby aiming for posture estimation with higher accuracy than when a single index is used.

Therefore, as shown in (4) and (5) of FIG. 10C, assuming that the geometric feature is the parallelogram according to the second embodiment described above (or one that can be used equivalently), A projective transformation matrix H _2P that returns to the shape can be obtained, but since its posture is unknown, this alone does not reveal the arrangement in the reference coordinate system.

For this reason, in the present invention, as shown in (3) and (4) of FIG.10D, a series of processes of the initial arrangement estimation unit 7 restores the parallelogram by the matrix _H2P and geometric features in the reference coordinate system. A matrix H ₂ ′ capable of arranging ₂ is obtained and arranged. Particularly in the present invention, it should be noted that the arrangement of the geometric feature 2 with respect to the index 1 is completely arbitrary and the degree of freedom of the form that the geometric feature 2 can take.

Then, by obtaining a planar projective transformation matrix as shown in (3) of FIG. 10E by the initial arrangement of the index 1 and the geometric feature 2 arranged in the common reference coordinate system, the posture estimation of the reference coordinate system can be performed using the index 1 It becomes possible to carry out with higher precision than the case of using only. Here, as shown in (4), the index 1 is the matrix H ₁ and the geometric feature 2 is determined in the reference coordinate system by the matrix H ₂ ′. The same applies to the case where the geometric feature 2 relates to the first embodiment.

  Various supplementary explanations will be given below.

  As a supplement to the first point, parallel or perpendicular line segments may be used that exist in a plurality of geometric shapes in the same plane of the reference coordinate system. That is, when line segment 1 is detected from geometric shape A and line segment 2 is detected from geometric shape B, it is known that line segment 1 and line segment 2 are parallel in the same plane of the reference coordinate system. For example, line segment 1 and line segment 2 may be counted as a set of parallel lines on this plane.

  For example, as shown in FIG. 11, when two geometric shapes of a square and a right triangle exist on the same plane of the reference coordinate system, line segments AB, DC, and GF that are parallel to this plane, There are line segments AD, BC, and EG that form parallel lines, line segments DA and DC that form right angles, line segments AC and BD that also form right angles, and GE and GF that also form right angles. doing. At this time, how to choose two sets of parallel lines,

[1] Parallel lines (AB, DC) and parallel lines (AD, BC); [2] Parallel lines (AB, DC) and parallel lines (AD, EG); [3] Parallel lines (AB, DC) And parallel lines (BC, EG),
[4] Parallel lines (AB, GF) and parallel lines (AD, BC), [5] Parallel lines (AB, GF) and parallel lines (AD, EG), [6] Parallel lines (AB, GF) And parallel lines (BC, EG),
[7] Parallel lines (DC, GF) and parallel lines (AD, BC); [8] Parallel lines (DC, GF) and parallel lines (AD, EG); [9] Parallel lines (DC, GF) And parallel lines (BC, EG),

  Nine ways listed in [1] to [9] are possible. Similarly, there are two ways of selecting two sets of orthogonal lines: orthogonal lines (DA, DC) and (AC, BD) and orthogonal lines (GE, GF) and (AC, BD).

  Note that orthogonal lines (DA, DC) and orthogonal lines (GE, GF) are parallel to each other and cannot be combined. In addition, for example, when the line segment DC and the line segment GF are in a straight line, they are treated as the same line segment, so that the number of combinations of parallel lines is reduced from nine to three.

In this example, it is possible to remove the affine distortion using two H _'A after the matrix configuration unit 72 nine in the H' _P is Motomari, the H 'to remove perspective distortion using _P, H combination of _'P and H' _a is present 18 kinds estimates respectively an initial placement of the geometric features using H _'P and H' _a of the 18 types.

For example, parallel lines (AB, DC) and (AD, BC) 'the _P, orthogonal lines (DA, DC) and (AC, BD) H from' to H when calculating the _A is, H _'P and H' The initial arrangement of point A, point B, point C, and point D can be estimated using H ′ configured using _A.

  The estimation calculation of the initial arrangement of geometric features may be performed for all 18 patterns in the above example. When estimation calculation of multiple initial arrangements is performed by all or selection explained below, select the case where the calculation error of the planar projective transformation matrix is finally minimized when estimating the posture in the reference coordinate system estimation unit 8 Alternatively, as described above, one may be selected by receiving an instruction from the user by pointing or the like.

In the estimation calculation, H ′ _P and H ′ _A may be preliminarily selected in the geometric feature detection unit 6 according to a certain standard, and then the estimation calculation may be performed in the initial arrangement estimation unit 7. . The constant reference, for example, when it comes to H _A, adopts an angle of orthogonal lines to each other used in the calculation of H _A as a kind of measure of the reliability of the H _A, the angle is largest or larger side A predetermined number may be selected.

Similarly with respect to H _P, may be selected a predetermined number of the largest ones or the side of higher angle parallel lines between the calculation time of the rectangle.

  Note that “the angle between orthogonal lines” and “the angle between parallel lines” express two sets of orthogonal lines or parallel lines as (line A1, line A2) and (line B1, line B2) (ie, “A1 ⊥A2, B1⊥B2 ”or“ A1‖A2, B1‖B2 ”) and (1) the angle between A1 and B1, (2) the angle between A1 and B2, (3) the angle between A2 and B1, (4) It can be obtained as the minimum value of the angles (1) to (4) (which take a value in the range of 0 ° to 90 °) consisting of the angle formed by A2 and B2.

  As a supplement to the second point, the initial arrangement estimation unit 7 obtains an initial arrangement of geometric features that do not exist on the same plane as the index, and is additionally used when estimating the reference coordinate system in the reference coordinate system estimation unit 8 You can also. That is, the initial arrangement is described in the index reference coordinate system (XY plane coordinates of XYZ coordinates) and is not on the index plane (Z = 0 plane) and is extended to the solid direction side with the Z axis added It becomes.

  Details will be described with reference to (2) and (3) of FIG. In Fig. 1, two parallel lines and two orthogonal lines exist as geometric feature 2 on the same plane P1 as index 1 in the reference coordinate system XYZ of (2). It can be obtained as described above. Here, it should be noted that the initial arrangement of the geometric feature 3 on the Y′Z ′ plane shown in (3) can also be obtained.

  In order to obtain the initial placement of geometric feature 3 on the Y'Z 'plane, a plane projective transformation matrix for the index on Y'Z' plane P2 is required, but index 1 is the same as Y'Z 'plane P2. It exists on another XY plane P1. Therefore, it is considered to convert the plane projective transformation matrix obtained based on the index 1 on the XY plane P1 into that on the Y′Z ′ plane. This can be realized by applying rotation and translation of the coordinate axes to the planar projective transformation matrix based on the relative relationship between the XYZ coordinate axes and the X′Y′Z ′ coordinate axes. Details regarding the calculation for applying rotation and translation of coordinate axes to the planar projective transformation matrix are disclosed in Patent Document 2.

  That is, in order to obtain the initial arrangement of the geometric feature 3 that does not exist on the same plane as the index 1, it is only necessary to have information that can identify the plane on which the geometric feature 3 exists. For example, the relative relationship between the XYZ coordinate axis and the X'Y'Z 'coordinate axis corresponds to the information to be specified, and when determining the reference coordinate system XYZ of the index 1, the X'Y'Z' coordinate axis is set in the reference coordinate system XYZ. Information to be identified may be added and held in the image processing apparatus 2 in advance.

  In the case of the rectangular parallelepiped in FIG. 2, if the rectangular information of the upper surface S1-S8-S2-S7 on which the index 1 is arranged in the reference coordinate system of the index 1 is previously stored, it is orthogonal because it is a rectangular parallelepiped. The positional relationship with respect to the upper surface of the other surfaces forming the relationship becomes known, and the initial arrangement of geometric features on the other surfaces can be obtained and added to the posture estimation calculation of the reference coordinate system. With this information, dimension measurement by the dimension measurement unit 9 for other surfaces can be performed in exactly the same manner.

  By using the geometric feature on the other surface, for example, the geometric feature 4 on the cylinder as shown in FIG. 12 or the geometric feature 5 on the two cones whose angles between the bottom surface and the side surface are known as shown in FIG. It is possible. In the case of the cylinder in FIG. 12, line segments (P41, P43) and (P42, P44) in the height direction of the cylinder can be detected, and the rectangles P41-P42-P43-P44 can be used as geometric features. In the case of the two cones in FIG. 13, the line segments (P51, P53) and (P52, P54) corresponding to the ridgeline can be detected, and the rectangle P51-P52-P53-P54 can be used as a geometric feature as well. . Here, the rectangle P41-P42-P43-P44 and the rectangle P51-P52-P53-P54 are figures on a plane that are conceptually configured on the inside of the cylinder and the side of the cone, respectively, due to the presence of the cylinder and the cone, There is no need to have an entity such as a wall.

  In the example of FIG. 1, the plane P1 and P2 identification processing also uses the pointing instruction from the user to the geometric feature when the geometric feature is detected, and information about which plane the geometric feature belongs to May be obtained from the user. As an automatic detection method, specific geometric features that cannot be detected anywhere other than the target plane are prepared in advance by distinguishing color features, etc., and the information is stored in the database 12 or the image processing device 2 In addition, among the indicators, those used for the plane P1 and those used for the plane P2 may be distinguished in advance.

  Similarly, as an automatic detection method, texture information and the like of the plane P1 and the plane P2 itself are stored, and the index detection unit 4 detects the planes P1 and P2 as areas in the captured image as preprocessing, The index may be detected by. You may make it make a user input the designation | designated information of the said plane area | region. Alternatively, the captured image itself may be prepared in a state where only the plane P1 or only the plane P1 and the plane P2 are captured in advance.

  As a third supplement, an embodiment using a plurality of indices will be described. FIG. 14 is a functional block diagram of the image processing apparatus 2 in the embodiment. The image processing apparatus 2 has a configuration in which another index initial arrangement estimation unit 15 is additionally provided in the configuration of FIG. The processing itself of each function unit common to FIG. 3 other than the other index initial arrangement estimation unit 15 is the same as in the case of the embodiment having one index (referred to as “single index embodiment”). However, as will be described below, since there are a plurality of indexes to be processed, each process is performed after distinguishing the indexes.

  In this embodiment, the index detection unit 4 detects a plurality of indexes from the captured image and passes them to the initial posture calculation unit 5. Then, one of the plurality of indices is set as a reference index, and for each index other than the reference index (referred to as “other index”), the initial allocation in the reference coordinate system defined by the reference index is determined by the other index initial layout estimation unit 15. Ask for. For this reason, the matrix calculation unit 51 and the matrix decomposition unit 52 perform the same processing as that of the single index embodiment for each of a plurality of indexes each including the reference index and the other indexes.

  Each of the plurality of indices is the same as that described with reference to FIG. 4, and is known in advance, and its arrangement is given in a known reference coordinate system. Note that the plurality of indices may include the same type. The plurality of indexes in the other embodiment are arranged on the same plane, and each of them is known, but the arrangement relationship between them is arbitrary on the same plane, and there is no restriction on the arrangement relationship with each other.

  As a specific example of such a plurality of indicators, for example, assuming a desk surface as the same plane, each printed on a card has a known marker freely arranged on the desk surface (however, they overlap each other) ).

Note that the estimation of the reference coordinate system using a plurality of indices including a reference index and other indices is shown in the following Patent Document 3 by the present inventors. The processes of the initial posture calculation unit 5 and the other index initial arrangement estimation unit 15 in the embodiment are the same as the processes shown in Patent Document 3.
[Patent Document 3] Japanese Patent Application No. 2012-174321 (“Image Processing Apparatus and Method”)

  In addition, as to which of a plurality of indicators is used as a reference indicator, any one of those detected by the indicator detection unit 4 using a random number or the like may be used. It may be determined in response to a user pointing instruction, a specific type of index may be used, or the area (area occupied by the index) in the captured image may be the maximum, or other It may be determined according to a predetermined standard.

  Further, the initial arrangement estimation unit 7 performs the same processing as in the single index embodiment, using the decomposition result of the plane projective transformation corresponding to the initial posture of the reference index in the matrix decomposition unit 52. The reference coordinate system estimation unit 8 is based on the overall correspondence relationship between the initial placement of the reference indices and other indices and geometric features in the reference coordinate system, and the placement of the reference indices and other indices and geometric features on the captured image. The posture of the reference coordinate system defined by the index with respect to the imaging device 10 is estimated.

  Here, unlike the case of the single index embodiment, the other correspondence index is added to the corresponding relationship to be used, but the posture estimation calculation itself from the corresponding relationship is the same as in the case of the single index embodiment. is there. In the other embodiment, by using another index in addition, the effect of further improving the posture estimation accuracy can be obtained in the same manner as the effect of using the geometric feature in the single index embodiment.

  Note that the differences and common points of the processing target data flow in this embodiment from the single index embodiment are as follows. That is, the line L1 shown in FIG. 14 corresponds to a single index embodiment that is implemented for the reference index. Lines L2, L3, and L4 correspond to the data flow newly generated for other indicators in this embodiment. Other flows are the same as those of the single index embodiment.

  The process in the other index initial arrangement estimating unit 15 corresponds to the process in which the initial arrangement estimating unit 7 obtains the initial arrangement of the geometric feature in the reference coordinate system defined by the reference index for the other index instead of the geometric feature. . By this processing, the initial arrangement of other indices is obtained in the reference coordinate system defined by the reference indices. Details of the processing are as follows.

As preconditions, the initial posture calculation unit 5 obtains a planar projection transformation matrix representing the initial posture and a decomposition result thereof for each of a plurality of indices. Therefore, following the form of (Equation 6), the following (Equation 21) with respect to the reference index is as follows with respect to the other indices (the index i for distinguishing each is used as another index i): (Equation 22) is obtained.
H ^-1 _{[reference index]} = H _{S [reference index]} H _{A [reference index]} H _{P [reference index]} (Equation 21)
H ^-1 _{[other index i]} = H _{S [other index i]} H _{A [other index i]} H _{P [other index i]} (Formula 22)

The other index initial arrangement estimation unit 15 uses the decomposition results for each index given by the initial posture calculation unit 5 as (Equation 21) and (Equation 22) above, and uses the other index i in the reference coordinate system defined by the reference index. A matrix H ′ _{[other index i]} for _obtaining the initial arrangement of is obtained as follows (Equation 23).
H ' _{[other index i]} = H _{S [standard index / other index i scale]} H _{A [other index i]} H _{P [other index i]} (Equation 23)
Here, in (Expression 23), H _{S [reference index / scale of other index i]} is the scale s = s when H _{S [reference index]} of (Expression 21) is expressed as (Expression 2) _{The [reference index]} is replaced with the scale s = s _{[other index i]} when H _{S [other index i]} in (Expression 22) is similarly expressed as (Expression 2).

Therefore, (Equation 23) used to obtain the initial arrangement of the other index i in the reference coordinate system of the _{reference index} is the scale of the position and orientation elements (HS _{[reference index]} ) of the initial posture calculated for the _{reference index.} of the initial posture calculated for the other index i (corresponding to replacing the scale s not to be considered with the scale s of H _{S [other index i]} ) and distortion (The H _{A [other index i]} representing the affine distortion and projective distortion, equivalent to _{HP [other index i]} ).

The other index initial arrangement estimating unit 15 finally sets the point x ′ (that is, the initial arrangement) of the other index i in the reference coordinate system defined by the reference index to the point x of the other index i on the captured image. Using (Equation 23) as a conversion coefficient, it is calculated from (Equation 24) below, and the initial arrangement x ′ is passed to the reference coordinate system estimation unit 8.
x '= H' _{[Other index i]} x ... (Equation 24)

  FIG. 15 is a diagram for explaining the significance of the other embodiment, and the drawn index 1 and geometric feature 2 and (1) to (4) are the same as the example of FIG. 10A. Here, index 2 is further added. Assume that index 1 is a reference index, index 2 is another index, and both are square indices. In this example, the other index is only one index 2, but a plurality of other indices may be present and may not be a square index.

In FIG. 15, as shown in (3), for index 1 as the reference index, the arrangement in the captured image system of (1) and the known initial arrangement in the reference coordinate system of the index 1 in (2) A matrix H ₁ is obtained that associates and gives an initial posture. As shown in (4) with the matrix H ₁ , converting the geometric feature 2 to the reference coordinate system of (2) does not necessarily return to the original shape, but in the matrix H ₁ to (5) As shown, even if the index 2 which is another index is converted into the reference coordinate system of (2), it does not necessarily return to the original square. This is because the mutual arrangement relationship between the index 1 and the index 2 existing on the same plane XY in (1) is unknown, and the matrix H ₁ is obtained only from the index 1.

Here, by applying the above (Equation 24), the index 2 can be returned to the original square in the reference coordinate system of the index 1 as shown in (6). Here, substantially the same manner as in the case of geometric features, to return to the square by correcting H _A strain did not return to the square _{Other indicators i]} H _{P [other indices i]} is in (2) (Formula 24) H _{S [reference index / scale of other index i]} acts as a conversion of index 1 to the reference coordinate system. Therefore, the arrangement of the index 2 as the square can be used as the initial arrangement of the index 2 in accordance with the initial arrangement that can be used when the mutual arrangement relationship between the index 1 and the index 2 is known.

Thus, finally in the reference coordinate system estimation unit 8, as shown in (6), in the reference coordinate system (by adopting the index 1 which is the reference index as a single index), the same as the embodiment of the single index The initial arrangement of the geometric feature 2 obtained in the above, the initial arrangement of the index 2 obtained in (Equation 24), the initial arrangement of the known index 1 in advance, and these captured image systems shown in (1) The matrix H shown in (7) is obtained as a correspondence relationship with the arrangement at, and the result is the posture estimation result of the reference coordinate system of index 1. The accuracy of the estimation result as the matrix H is improved by the additional use of the geometric feature 2 and the index 2 that is another index than the matrix H ₁ estimated only from the index 1 that is the reference index.

  DESCRIPTION OF SYMBOLS 1 ... AR system, 2 ... Image processing apparatus, 10 ... Imaging apparatus, 11 ... Information display apparatus, 3 ... Captured image acquisition part, 4 ... Index | index detection part, 5 ... Initial arrangement | positioning calculation part, 6 ... Geometric feature detection part, 7 ... initial placement estimation unit, 8 ... reference coordinate system estimation unit, 9 ... dimension measurement unit, 15 ... other indicator initial placement estimation unit

Claims (9)

A plane on which an index configured to identify at least four points or line segments on the same plane, which is a predetermined index whose size and arrangement are known, from a captured image captured by the imaging device An image processing apparatus that estimates the reference coordinate system defined by the index using the index and a predetermined geometric feature configured as a parallel relationship and / or an orthogonal relationship on the plane,
An index detection unit for detecting the index from the captured image;
Calculating an initial posture of the reference coordinate system with respect to the imaging apparatus based on a predetermined initial arrangement in which the index is expressed in the reference coordinate system and an arrangement of the detected index on the captured image; An initial posture calculation unit that decomposes the transformation relationship represented by the initial posture into elements corresponding to the size, position, orientation, and distortion;
A geometric feature detector for detecting the geometric feature from the captured image;
An initial arrangement estimating unit that estimates an initial arrangement of the geometric feature in the reference coordinate system based on the detected distortion element in the geometric feature and the decomposed size, position, and orientation elements;
Based on the overall correspondence of the predetermined initial arrangement of the index and the initial arrangement of the estimated geometric feature and the arrangement of the detected index and geometric feature on the captured image, the reference coordinate system An image processing apparatus comprising: a reference coordinate system estimation unit that estimates a posture with respect to the imaging apparatus.   The initial posture calculation unit obtains the initial posture as a relationship of planar projective transformation, and decomposes the relationship into a similarity transformation corresponding to the size, position, and orientation, and an affine transformation and a projective transformation corresponding to distortion. 2. The image processing apparatus according to claim 1, wherein: The predetermined geometric features are configured as parallel and orthogonal relationships on the plane;
The initial arrangement estimating unit extracts a projection distortion element from a parallel relation distortion in the detected geometric feature, further extracts an affine distortion element from an orthogonal relation distortion, and the extracted projection distortion and affine distortion. 3. The image processing according to claim 2, wherein an initial arrangement of the geometric feature in the reference coordinate system is estimated based on a conversion relationship determined by a combination of the element and a similarity conversion element among the decomposed elements. apparatus. The predetermined geometric feature is configured as a parallel relationship on the plane;
The initial arrangement estimation unit extracts projection distortion elements from parallel distortions in the detected geometric features, the extracted projection distortion elements, and the elements of similarity decomposition and affine transformation among the decomposed elements 3. The image processing apparatus according to claim 2, wherein an initial arrangement of the geometric feature in the reference coordinate system is estimated based on a conversion relationship determined by combining the two.   When the geometric feature detecting unit detects a plurality of parallel relationships and / or orthogonal relationships in the predetermined geometric feature, the geometric feature detecting unit is based on the magnitude of the angle between line segments forming the parallel relationship and / or the orthogonal relationship on the plane. 5. The image processing apparatus according to claim 1, wherein a selection is made from among the plurality, and the initial arrangement estimation unit performs the estimation for the selected geometric feature.   2. The apparatus according to claim 1, further comprising a dimension measuring unit that measures a dimension of a predetermined location on the captured image based on the estimated orientation of the reference coordinate system and a known size of the predetermined index. 6. The image processing device according to any one of 5 to 5. The predetermined geometric feature is further a plane different from the plane, and is also installed on another plane whose posture in a reference coordinate system defined by the predetermined index is known,
The initial arrangement estimating unit calculates an initial arrangement of the geometric feature on the different plane in the reference coordinate system, a distortion element in the geometric feature detected on the different plane, the decomposed size, position, and orientation. 7. The image processing apparatus according to claim 1, wherein the estimation is performed by further adding an element of the known posture in addition to the element. There are a plurality of the indicators, and the indicator detection unit detects a plurality of indicators from the captured image,
The initial posture calculation unit calculates the initial posture for each index, and decomposes the transformation relationship represented by each initial posture into elements corresponding to magnitude, position, orientation, and distortion,
Using one of the plurality of existing indices as a reference index for defining the reference coordinate system, the initial arrangement estimating unit is decomposed with respect to a distortion element in the detected geometric feature and the reference index as a target. Estimating the initial placement of the geometric feature in the reference coordinate system based on the size and position and orientation elements;
For each of the other indices other than the reference index, based on the decomposed size and distortion elements with respect to the other indices, and the decomposed position and orientation elements with respect to the reference index, Further comprising another index initial arrangement estimating unit for estimating an initial arrangement of the other index in the reference coordinate system,
The reference coordinate system estimation unit includes a predetermined initial arrangement of the reference index and an initial arrangement of the estimated other index and a geometric feature, and an arrangement of the detected reference index, the other index and a geometric feature on the captured image. The image processing apparatus according to claim 1, wherein an attitude of the reference coordinate system with respect to the imaging apparatus is estimated based on a correspondence relationship as a whole. A plane on which an index configured to identify at least four points or line segments on the same plane, which is a predetermined index whose size and arrangement are known, from a captured image captured by the imaging device An image processing method for estimating a reference coordinate system defined by the index using the index and a predetermined geometric feature configured as a parallel relationship and / or an orthogonal relationship on the plane,
A detection step of detecting the index and the geometric feature from the captured image;
Calculating an initial posture of the reference coordinate system with respect to the imaging apparatus based on a predetermined initial arrangement in which the index is expressed in the reference coordinate system and an arrangement of the detected index on the captured image; An initial posture calculation step for decomposing the transformation relationship represented by the initial posture into elements corresponding to the size, position and orientation, and distortion;
An initial placement estimating step for estimating an initial placement of the geometric feature in the reference coordinate system based on the distortion element in the detected geometric feature and the decomposed size and position and orientation elements;
Based on the overall correspondence of the predetermined initial arrangement of the index and the initial arrangement of the estimated geometric feature and the arrangement of the detected index and geometric feature on the captured image, the reference coordinate system An image processing method comprising: a reference coordinate system estimation step for estimating a posture with respect to the imaging apparatus.