JP2019105992A - Image processing device, image processing program and image processing method - Google Patents

Abstract

To reduce a calculation amount of determining a pair of a feature line of a target object in a photographic image used to estimate the position and attitude of an imaging position and a line segment of shape information representing the shape of a corresponding object.SOLUTION: An image processing device includes: a three-dimensional line segment extraction part for extracting a plurality of three-dimensional line segments from three-dimensional shape information; a feature line extraction part for extracting a plurality of feature lines from a photographic image; a pair estimation part for estimating one pair of a first feature line and a first three-dimensional line segment on the basis of a distance between a first distribution about angles of the plurality of feature lines and a second distribution about angles of a plurality of projection lines obtained by projecting a plurality of three-dimensional line segments to a projection surface, and estimating at least three pairs of the feature lines and the three-dimensional line segments on the basis of the estimated one pair; and a viewpoint estimation part for estimating the position and the direction of an imaging device with respect to a target object in a three-dimensional space from the photographic image on the basis of at least four estimated pairs.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image processing apparatus, an image processing program, and an image processing method.

  In recent years, systems that display images using Augmented Reality (AR) technology have become widespread (see, for example, Patent Documents 1 to 3). In an example of the AR technology, an object is photographed using a camera mounted on a personal computer (PC), a portable terminal device or the like, and the position and orientation of the camera in a three-dimensional space are estimated from an image of the object. Then, content information is superimposed and displayed at an arbitrary position in the image based on the determined position and orientation of the camera.

  As content information to be superimposed and displayed, for example, Computer-Aided Design (CAD) data representing a three-dimensional shape of an object is used. In order to determine the position and orientation of the camera, at least four pairs of three-dimensional line segments of CAD data and corresponding line segments of an object in a captured image are used.

  As a first technique, there is a technique for comparing a three-dimensional structure with a model represented by model information (for example, Patent Document 1). In the first technique, the computer can select an edge line extracted from a captured image obtained by capturing an image of a three-dimensional structure by an imaging device and a ridgeline included in a model image represented by model information of the three-dimensional structure. Display with. Next, the computer receives a selection instruction indicating an edge line and a ridge line to be superimposed. Then, in response to the received selection instruction, the computer displays a superimposed image in which the model image is superimposed on the captured image such that the edge line to be superimposed and the ridge line overlap.

  As a second technology, there is a technology for estimating the position of an imaging device using an image of an object captured by the imaging device and shape information (for example, Patent Document 2). In the second technique, the computer uses the predetermined number of combinations in which a projection line obtained by projecting a candidate line included in shape information of an object onto an image and a feature line detected from the image are associated with each other. Estimate the first position of Next, the computer generates a first projection line by projecting another candidate line onto the image using the first position, and selects a feature line corresponding thereto. Next, the computer estimates the second position of the imaging device using a combination in which the first projection line and the feature line are associated with each other. The computer then projects the further candidate line onto the image using the second position to generate a second projection line, which represents the deviation between the second projection line and the further feature line. The third position of the imaging device is determined based on the index.

JP 2017-91078 A JP, 2017-182302, A JP, 2015-118641, A

  When the position and orientation of the camera are estimated by associating the image of the object captured by the camera with the CAD data of the object, the amount of calculation for association is large.

  Such a problem arises not only in the case of associating the image of the object with the CAD data, but also in the case of associating the image of the object with other shape information.

  In one aspect, the present invention is for determining a pair of a feature line of an object in a captured image used for estimating the position and orientation of an imaging device and a line segment of shape information representing the shape of the corresponding object. The purpose is to reduce the amount of calculation.

  According to one aspect, the image processing apparatus includes a three-dimensional line segment extraction unit that extracts a plurality of three-dimensional line segments that form a three-dimensional shape from three-dimensional shape information indicating the three-dimensional shape of the object; A feature line extracting unit for extracting a plurality of feature lines indicating features of the shape of the object from a photographed image obtained by photographing the object; a first distribution indicating a distribution of angles of the plurality of feature lines; A first feature line of the plurality of feature lines based on a distance between the plurality of feature lines obtained by projecting the three-dimensional line segment onto the projection plane and a second distribution indicating a distribution of angles of the plurality of projection lines; While estimating one pair of a plurality of three-dimensional line segments with the first three-dimensional line segment, at least three pairs of feature lines and three-dimensional line segments are estimated based on the estimated pair of pairs. From the captured image, based on the pair estimation unit to be estimated and the estimated at least four pairs. Including a viewpoint estimation unit for estimating the position and orientation of the imaging device with respect to the object in the dimension space.

  According to the embodiment, the amount of calculation for determining a pair of a feature line of an object in a captured image used for estimating the position and orientation of an imaging device and a line segment of shape information representing the shape of the corresponding object It can be reduced.

It is a figure for demonstrating the method to obtain | require the position and attitude | position of a camera using the feature point contained in the image | photographed image. It is a figure which shows the example of the production | generation method of a three-dimensional map. It is a figure which shows the example of the picked-up image of an object. It is a figure which shows the example of CAD data. It is a figure which shows the example of matching with the edge line detected from the captured image, and the outline represented by CAD data. It is a figure showing an example of the image processing device in an embodiment. It is a flowchart which shows the example of the image processing in this embodiment. It is a figure which shows the example of detection of the feature line from the captured image in this embodiment. It is a figure which shows the example of a detection of the three-dimensional line segment in this embodiment. It is a figure for _{demonstrating} rotating a 3D model so that line segment L _mdl in this embodiment may become parallel to line segment L _img on a projection plane. It is a figure explaining the normal line n of the plane which line segment L _img in this embodiment and a camera viewpoint make. It is a figure for _{demonstrating} the state transition at the time of rotating a 3D model by making line segment L _mdl and each normal line n into an axis in this embodiment. FIG. 7 is a diagram showing an example of a feature line extracted from a photographed image and a three-dimensional line segment of a 3D model rotated based on a temporary corresponding line segment pair in the present embodiment. It is a figure which shows the example of the direction (angle) of the characteristic line on the picked-up image in this embodiment, and the direction (angle) of each projection line of 3D model in each rotation state. It is a figure which shows the example of the histogram about the angle of the feature line on the picked-up image in this embodiment, and the histogram about the angle of each projection line of 3D model in each rotation state. It is a figure for demonstrating the example of the distance between the histogram about the angle of the feature line on the picked-up image in this embodiment, and the histogram about the angle of each projection line of 3D model in each rotation state. It is a figure showing an example of the image processing device in this embodiment (example 1). It is a functional block diagram of the pair estimation part in this embodiment (Example 1). It is a flowchart which shows the specific example of the image processing in this embodiment (Example 1). It is a flowchart of detailed processing of estimation processing (step 1906) of the corresponding line segment pair in the present embodiment (Example 1). It is a flowchart of a detailed process of processing (step 2008) which presumes other 3 pairs of corresponding line segment pairs in this embodiment (Example 1). It is a figure which shows the example of the corresponding line segment pair in this embodiment (Example 1). It is a figure which shows the calculation method based on the area of the area | region in this embodiment (Example 1). It is a figure which shows the calculation method based on the distance in this embodiment (Example 1). It is a figure which shows the line segment rotated 180 degrees in this embodiment (Example 1). It is a figure which shows the example of the three-dimensional line segment which is not suitable for calculation of the rough | crude parameter in this embodiment (Example 1). It is a functional block diagram of the pair estimation part in this embodiment (Example 2). It is a flowchart which shows the specific example of the image processing in this embodiment (Example 2). It is a detailed flowchart of feature line selection (step 2001) in the present embodiment (Example 2). It is a figure explaining the method to select from the outside feature line in this embodiment (Example 2). It is a flowchart of detailed processing of three-dimensional line segment selection (step 2002) in the present embodiment (Example 2). It is a figure for demonstrating the method to select the corresponding line segment in this embodiment (Example 2). It is a figure which shows the structural example of the information processing apparatus (computer) used as an image processing apparatus in this embodiment (Example 1, Example 2).

Hereinafter, embodiments will be described in detail with reference to the drawings.
FIG. 1 is a diagram for explaining a method of determining the position and orientation of a camera using feature points included in a captured image. In this method, a three-dimensional map 101 representing a set of three-dimensional coordinates of map points (positions in three dimensions) 111 on an object is generated in advance.

  Next, when the image 102 is captured by the camera, the map points 111 are projected onto the image 102 as a projection plane using a transformation matrix M that transforms the three-dimensional coordinate system 103 into the camera coordinate system 104. A projection point 112 is determined. Then, by associating the projection point 112 with the feature point 113 detected from the image 102, the position and orientation of the camera in the three-dimensional coordinate system 103 are estimated. The attitude of the camera means the direction of the viewpoint of the camera.

  For example, the position of the camera capturing the image 102 is represented by the relative position of the camera coordinate system 104 with respect to the three-dimensional coordinate system 103, and the posture of the camera is represented by the relative angle of the camera coordinate system 104 with respect to the three-dimensional coordinate system 103. Ru.

  The three-dimensional coordinate Sp of the map point p, the two-dimensional coordinate xp 'of the projection point corresponding to the map point p, and the two-dimensional coordinate xp of the feature point corresponding to the map point p can be described as .

Sp = (x, y, z) (1)
xp '= (u', v ') (2)
xp = (u, v) (3)

  In this case, the square sum E of the distance between the projection point and the feature point on the image is expressed by the following equation.

  The position and orientation of the camera can be determined by determining a conversion matrix M that minimizes the sum of squares E in equation (4).

  Further, a line segment connecting the projection points 112 is referred to as a projection line 115, and the vector thereof is referred to as a direction vector of the projection line.

  FIG. 2 is a diagram showing an example of a method of generating the three-dimensional map 101. As shown in FIG. In this generation method, stereo photography and stereo measurement are used. The image 201 and the image 202 photographed respectively from the photographing position 211 and the photographing position 212 are used as key frames, and by associating the feature points 213 in the image 201 with the feature points 214 in the image 202, a map in a three-dimensional space Point 215 is restored. By associating a plurality of feature points in two images, a plurality of map points are restored, and a three-dimensional map 101 representing a set of the map points is generated.

  When CAD data of an object is used as the three-dimensional map 101 of FIG. 1 when determining the position and orientation of a camera, the “look” characteristics such as pattern or color often differ between the CAD data and the object. . It is difficult to automatically associate the feature points in the image with the map points represented by the CAD data by image processing on the assumption that the features of the "look" match. For this reason, the correspondence between the feature points and the map points may be manually performed by the mouse operation on the screen of the PC. In this case, the following problems are considered to occur.

(A) It is difficult to accurately detect feature points in an image.
(B) It is difficult to select a projection point and a feature point by projecting a map point onto an image with a mouse or the like.
(C) The number of combinations (corresponding pairs) of the projection point and the feature point to be selected is limited due to the difficulty in the operation of selecting the projection point and the feature point and the labor of manual work.
(D) Since the number of corresponding pairs is small, the calculation accuracy of the position and orientation of the camera is reduced.

  Therefore, as described in Patent Document 1, a method may be considered in which edge lines detected from an image are associated with outlines represented by CAD data.

  FIG. 3 is a view showing an example of a photographed image of an object. The captured image of FIG. 3 is an image obtained by capturing an object as a subject (target object). FIG. 4 is a diagram showing an example of CAD data. The CAD data of FIG. 4 is CAD data corresponding to the object of FIG.

  FIG. 5 is a diagram showing an example of the correspondence between edge lines detected from a captured image and outlines represented by CAD data. Hereinafter, the association of the photographed image of FIG. 3 with the CAD data of FIG. 4 will be described.

  First, as shown in FIG. 5A, an image processing apparatus such as a portable terminal apparatus performs edge detection processing to detect an edge line from the photographed image of FIG. Next, as shown in FIG. 5B, the image processing apparatus displays an outline represented by CAD data and a detected edge line in the captured image. The user associates the photographed image edge lines and the outlines with each other by selecting them with a mouse or the like. In order to obtain the position and orientation (direction of the camera viewpoint) of the camera that captured the captured image, it is desirable to use at least four corresponding pairs.

  Next, the image processing apparatus calculates the position and orientation of the camera that captured the captured image, using the combination of the associated edge line and outline. Then, as shown in FIG. 5C, the image processing apparatus superimposes and displays an outline represented by CAD data on the image of the object in accordance with the calculated position and orientation of the camera.

  According to such a correspondence method, edge lines can be detected with high accuracy, and edge lines and outlines can be easily selected, thereby solving the problems (A) and (B). However, it is considered that the following problems occur.

(E) The number of combinations (corresponding pairs) of contour lines and edge lines to be selected is limited due to the time and labor of manual work.
(F) Since the number of corresponding pairs is small, the calculation accuracy of the position and orientation of the camera is reduced.

  Therefore, as described in Patent Document 2, a method is also conceivable in which a correspondence pair is automatically added to recalculate the position and orientation of the camera. The calculation accuracy is improved by increasing the number of corresponding pairs. However, also in this method, a predetermined number of initial corresponding pairs are selected by the user, and a specific method of automatically selecting an initial corresponding pair is not described.

  When a non-expert user manually selects a corresponding pair visually, the operation takes time and a selection error may occur.

On the other hand, a method of performing round-robin calculation using all combinations of edge lines detected from an image and contour lines represented by CAD data is also conceivable. For example, when four edge lines are selected from m edge lines and four outline lines are selected from n outline lines to generate four corresponding pairs, the total number of combinations is _m C ₄ × _n C ₄ pieces. Therefore, as the number of edge lines and the number of contour lines increase, the total number of combinations increases, and the amount of calculation for generating corresponding pairs increases.

  In the present embodiment, as an example, when three-dimensional shape information such as a 3D model or CAD data is present and an approximate shooting distance is known, in the superposition of the 3D model on the shot image, selection of corresponding line segment pairs Is automated to estimate the position and orientation of the camera. In this way, the time and effort of manual work can be reduced, and the processing time can be shortened and errors can be reduced.

  FIG. 6 is a diagram illustrating an example of the image processing apparatus according to the embodiment. The image processing device 601 includes a three-dimensional line segment extraction unit 602, a feature line extraction unit 603, a pair estimation unit 604, and a viewpoint estimation unit 605.

  The three-dimensional line segment extraction unit 602 extracts a plurality of three-dimensional line segments forming a three-dimensional shape from three-dimensional shape information indicating the three-dimensional shape of the object. An example of the three-dimensional line segment extraction unit 602 is a three-dimensional line segment extraction unit 1703 described later.

  The feature line extraction unit 603 extracts a plurality of feature lines indicating features of the shape of the object from the photographed image obtained by photographing the object by the imaging device. An example of the feature line extraction unit 603 is a feature line detection unit 1705 described later.

  The pair estimation unit 604 determines, based on the distance between the first distribution and the second distribution, the first feature line of the plurality of feature lines and the first of the plurality of three-dimensional line segments. Estimate a pair of 3D line segments. The first distribution shows the distribution of the angles of the plurality of feature lines. The second distribution indicates a distribution of angles of a plurality of projection lines obtained by projecting a plurality of three-dimensional line segments on a projection plane. The pair estimation unit 604 estimates at least three pairs of a feature line and a three-dimensional line segment based on the estimated pair. An example of the pair estimation unit 604 is an estimation unit 1706 described later.

  The viewpoint estimation unit 605 estimates the position and direction of the imaging device with respect to the object in the three-dimensional space from the captured image based on the estimated at least four pairs. Examples of the viewpoint estimation unit 605 include a position calculation unit 1707, an error calculation unit 1708, and a determination unit 1709, which will be described later.

  With this configuration, calculation is performed to determine a pair of a feature line of an object in a captured image used to estimate the position and orientation of the imaging device and a line segment of shape information representing the shape of the corresponding object. The amount can be reduced.

  The pair estimation unit 604 selects a first feature line from the plurality of extracted feature lines, and selects a first three-dimensional line segment from the plurality of extracted three-dimensional line segments. The pair estimation unit 604 rotates the three-dimensional shape in the virtual space to match the first feature line with the first three-dimensional line segment. The pair estimation unit 604 has a three-dimensional shape in which the first feature line matches the first three-dimensional line segment, the first three-dimensional line segment, the first feature line, and the viewpoint of the imaging device. The three-dimensional shape is rotated by a predetermined angle with the normal to the plane to be made as the rotation axis. The pair estimation unit 604 generates a second distribution of the angles of the plurality of projection lines obtained by projecting the plurality of three-dimensional line segments on the projection plane each time the pair of rotations is rotated. The pair estimation unit 604 estimates a pair of pairs according to the distance between each of the plurality of generated second distributions and the first distribution.

  With such a configuration, it can be estimated whether the feature line selected from the photographed image and the three-dimensional line segment selected from the three-dimensional shape information correspond to one another.

  When the pair estimation unit 604 estimates a first pair that is a pair of pairs, one from a plurality of projection lines other than the first projection line of the first three-dimensional line segment that forms the first pair. A second projection line whose angle coincides with a second feature line other than the first feature line forming the pair is acquired. The pair estimation unit 604 selects the second three-dimensional line segment and the second feature line corresponding to the second projection line, and generates a second pair of the second three-dimensional line segment and the second feature line. Estimate.

  From the plurality of projection lines other than the first projection line and the second projection line, the pair estimation unit 604 coincides in position and angle with the third feature line other than the first feature line and the second feature line. Acquire a third projection line. The pair estimation unit 604 selects the third three-dimensional line segment and the third feature line corresponding to the third projection line, and the third three-dimensional line segment and the third feature line as a third pair Estimate.

  The pair estimation unit 604 determines, from the plurality of projection lines other than the first projection line, the second projection line and the third projection line, the first feature line, the second feature line and the third feature line. A fourth projection line whose angle and position match the fourth feature line is acquired. The pair estimation unit 604 selects the fourth three-dimensional line segment and the fourth feature line corresponding to the fourth projection line, and selects the fourth three-dimensional line segment and the fourth feature line as the fourth pair. presume.

  By this configuration, when the first pair is estimated, the first pair is temporarily fixed, and another pair is determined from the relationship between the position and angle of another feature line and a three-dimensional line segment. It can be estimated.

  The pair estimation unit 604 groups a plurality of feature lines extracted from the captured image in the same direction to form a plurality of feature line groups. The pair estimation unit 604 groups a plurality of three-dimensional line segments in the same direction to form a plurality of three-dimensional line segment groups.

  The pair estimation unit 604 sets at least one condition among the number of feature lines included in each feature line group, the length of feature lines included in each feature line group, and the position of feature lines in each feature line group The first feature line is selected based on

  The pair estimation unit 604 is based on at least one condition among the number of 3D line segments included in each 3D line segment group and the length of 3D line segments included in each 3D line segment group. , Select a first three-dimensional line segment.

  With this configuration, it is possible to increase the expected value for obtaining the correct corresponding line segment pair when selecting a three-dimensional line segment, and to speed up the process of searching for the corresponding line segment pair.

  The pair estimation unit 604 groups a plurality of feature lines extracted from the captured image in the same direction to form a plurality of feature line groups. The pair estimation unit 604 groups a plurality of three-dimensional line segments in the same direction to form a plurality of three-dimensional line segment groups.

  The pair estimation unit 604 selects a feature line group including the largest number of feature lines from the plurality of feature line groups. When the lengths of all the feature lines included in the selected feature line group are within a predetermined range, the pair estimation unit 604 sets the feature line corresponding to the outer circumferential line in the selected feature line group as the first feature line. Choose as. When the length of any feature line included in the selected feature line group exceeds a predetermined range, the pair estimation unit 604 selects the longest feature line from the selected feature line group as the first feature line. .

  The pair estimation unit 604 selects one three-dimensional line segment group including a three-dimensional line segment whose number is equal to or more than the number of feature lines included in the selected feature line group from among a plurality of three-dimensional line segment groups. Selects one 3D line segment group. The pair estimation unit 604 selects the three-dimensional line segment group including the longest three-dimensional line segment, when there are a plurality of three-dimensional line segment groups including the three-dimensional line segment having a closer number by the number of feature lines or more. . When there is no three-dimensional line segment group including three or more three-dimensional line segments equal to or more than the number of feature lines, the pair estimation unit 604 has a closer number of three-dimensional lines less than the number of feature lines included in the selected feature line group. Select 3D line segment group including line segments. The pair estimation unit 604 selects the longest three-dimensional line segment as the first three-dimensional line segment from the selected three-dimensional line segment group.

  With this configuration, it is possible to increase the expected value for obtaining the correct corresponding line segment pair when selecting a three-dimensional line segment, and to speed up the process of searching for the corresponding line segment pair.

FIG. 7 is a flowchart showing an example of image processing in the present embodiment. FIG. 8 is a diagram showing an example of detection of feature lines from a captured image in the present embodiment. FIG. 9 is a diagram showing an example of detection of a three-dimensional line segment in the present embodiment. FIG. 10 illustrates that the 3D model is rotated so that the line segment L _mdl in this embodiment is parallel to the line segment L _img on the projection plane, that is, the angles of the temporary corresponding line segment pair coincide with each other. FIG.

  In FIG. 7, the image processing apparatus 601 simply estimates whether the line segment on the captured image and the line segment selected from the 3D model correspond to each other (corresponding line segment), and the corresponding line segments If it is estimated that there is no line segment in the 3D model. The image processing apparatus 601 estimates another set of corresponding line segments when the corresponding line segments are estimated, and performs precise calculation when a set of four or more pairs is found.

  As shown in FIG. 8A, the feature line extraction unit 603 detects a feature line (a line segment detected from a shot image) from a shot image shot by a camera or the like (step 701).

  The three-dimensional line segment extraction unit 602 detects three-dimensional line segments from three-dimensional shape information (hereinafter referred to as "3D model") such as 3D model data or CAD data as shown in FIG. 9A. (Step 702). The three-dimensional line segment is information indicating a line segment that forms a three-dimensional shape.

  As shown in FIG. 9B, the pair estimation unit 604 virtually arranges the camera at an appropriate position in a virtual space, and the 3D model at an approximate shooting distance on the line of sight of the camera ( Step 703).

The pair estimation unit 604 selects the line segment L _img (FIG. 8 (b)) on the captured image and the line segment L _mdl (FIG. 9 (b)) on the 3D model (step 704). The pair of the selected line segment L _img and the line segment L _mdl is referred to as a temporary corresponding line segment pair for convenience of description.

As shown in FIG. 10, the pair estimation unit 604 sets the 3D model so that the line segment L _mdl is parallel to the line segment L _img on the projection plane, that is, the angles of the temporary corresponding line segment pair coincide with each other. Rotate (step 705).

The pair estimation unit 604 obtains the direction vector of the projection line of the 3D model by rotating the 3D model with the line segment L _mdl as an axis and with the normal line n between the line segment L _img and the camera viewpoint as an axis. (Step 706). The direction vector of the projection line of the 3D model is represented by reference numeral 115 as described in FIG.

FIG. 11 is a diagram for explaining the normal line n of the plane formed by the line segment L _img and the camera viewpoint in the present embodiment. An image plane 1101 is a plane captured from a camera viewpoint (camera coordinate system origin) 1104. A line segment L _img exists on the image plane 1101. The line segment L _img corresponds to the edge 1102 on the 3D model. At this time, a line segment in the direction perpendicular to the plane formed by the line segment L _img and the camera viewpoint is indicated by the normal line n.

FIG. 12 is a diagram for explaining state transition in the case where the 3D model is rotated with the line segment L _mdl and the normal line n as axes in the present embodiment. In FIG. 12, the horizontal direction indicates the state transition of the 3D model when the 3D model is rotated about the selected three-dimensional line segment L _mdl . The vertical direction indicates the state transition of the 3D model in the case where the 3D model is rotated about the normal line n of the plane 1105 formed by the line segment L _img and the camera viewpoint.

As shown in FIG. 12, the 3D model is rotated about the line segment L _mdl and the normal line n, and the direction vector of the projection line of the 3D line segment forming the 3D model in the rotated state is determined.

  It returns to the explanation of FIG. The pair estimation unit 604 uses the distance between the histogram H1 of the direction (angle) of the feature line on the captured image and the histogram H2 of the direction (angle) of the projection line as an index It is determined whether or not there is (step 707). Step 707 will be described using FIGS. 13 to 16.

  FIG. 13 is a diagram illustrating an example of a feature line extracted from a photographed image and a three-dimensional line segment of a 3D model rotated based on a temporary corresponding line segment pair in the present embodiment. A feature line extracted from the photographed image is obtained in step 701.

  In step 707, the pair estimation unit 604 compares the feature line 1301 extracted from the photographed image with the 3D models 1302, 1303, and 1304 projected on the projection plane in each rotation state, and the temporary corresponding line segment pair corresponds. Determine if there is a possibility of a line segment. In this case, the pair estimation unit 604 uses the distance between the histogram H1 of the direction (angle) of the feature line on the captured image and the histogram H2 of the direction (angle) of the projection line as an index. Determine if there is a potential for minutes.

  First, the direction (angle) of the feature line on the photographed image and the direction (angle) of each projection line of the 3D model in each rotation state are calculated. The angle of the feature line on the captured image is the inclination of the feature line on the captured image, and is represented by, for example, the angle with respect to the X axis of the camera coordinate system 104 in FIG. The angle of the projection line is the inclination of the projection line in the projection plane, and is represented, for example, by the angle with respect to the X axis of the camera coordinate system 104 of FIG.

  FIG. 14 is a view showing an example of the direction (angle) of the feature line on the photographed image and the direction (angle) of each projection line of the 3D model in each rotation state in the present embodiment.

  FIG. 14A shows the directions (angles) of 15 feature lines on the captured image 1301. FIG. 14B shows the directions (angles) of 15 projection lines obtained when the 3D model 1302 formed by 15 three-dimensional line segments is projected onto the imaging plane (corresponding to the image 102 in FIG. 1). Indicates FIG. 14C shows the directions (angles) of 15 projection lines obtained when the 3D model 1303 formed of 15 three-dimensional line segments is projected onto the imaging plane (corresponding to the image 102 in FIG. 1). Indicates FIG. 14C shows the directions (angles) of 15 projection lines obtained when the 3D model 1304 formed by 15 three-dimensional line segments is projected onto the imaging plane (corresponding to the image 102 in FIG. 1). Indicates

  FIG. 15 is a diagram showing an example of a histogram of angles of feature lines on a photographed image and a histogram of angles of projection lines of the 3D model in each rotation state in the present embodiment. The horizontal axis of each of the histograms 1501, 1502, 1503, and 1504 indicates an angle (in 3 degree units), and the vertical axis indicates the number (frequency) of angles included in the range of the angle.

  The histogram H1 (1501) is a histogram of the angle of the feature line on the photographed image 1301, and is a histogram of FIG. 14 (a) in units of three degrees.

  The histogram H2 (1502) is a histogram of the angle of each projection line of the 3D model 1302, and is a histogram of FIG. 14 (b) in units of three degrees.

  The histogram H2 (1503) is a histogram of the angle of each projection line of the 3D model 1303, and is a histogram of FIG. 14C in units of three degrees.

  The histogram H2 (1504) is a histogram of the angle of each projection line of the 3D model 1304, and is a histogram of FIG. 14 (d) in units of three degrees.

  In the histogram, the unit of frequency aggregation is referred to as a bin (the column of the histogram graph), and as described above, the unit of bins and the number of bins of the histogram H1 and the histogram H2 are made the same.

  FIG. 16 is a view for explaining an example of the distance between the histogram H1 of the angle of the feature line on the photographed image and the histogram H2 of the angle of each projection line of the 3D model in each rotation state in this embodiment. FIG.

Here, how similar the histograms to be compared are to each other is referred to as a distance between the histograms, and hereinafter, it is represented by d (x ₁ , x ₂ ). x ₁ is an n-dimensional vector whose element is the frequency of each of n bins of the histogram H ₁ . x ₂ is an n-dimensional vector whose element is the frequency of each of n bins of the histogram H ₂ .

  The more similar the histograms are, the smaller the value of the distance between the histograms, and the different the histograms are, the larger the value of the distance between the histograms. Although various methods can be used to calculate the distance between histograms, an example of the calculation method is shown below.

Here, the histogram H1 and the histogram H2 are drawn in the same two-dimensional coordinate system. At this time, the coordinate of the top of each bin of the histogram H1 is represented by x _1i , and the coordinate of the top of each bin of the histogram H2 is represented by x _2i . x _1i is the i-th (i = 1 to n) element of the vector x ₁ . x _{2 i} is the i-th (i = 1 to n) element of the vector x ₂ .

・ Manhattan distance (also called L1 norm)
Manhattan distance d (x _1, x _2), as shown in Equation (5), the distance sum of the coordinates x _1i and the coordinate x _2i each histogram difference H2 of the histogram H1 (absolute value) between two points Can be represented by Here, Σ in equation (5) represents the total sum from i = 1 to i = n, and n is the number of bins of the histogram H1 and the histogram H2.

Euclidean distance (also called L2 norm)
Euclidean distance d (x _1, x _2), as shown in Equation (6) represents the square root of the sum of the squares of the coordinates x _1i and the coordinate x _2i each histogram difference H2 of the histogram H1 as the distance between two points be able to. Here, Σ in equation (6) represents the sum from i = 1 to i = n, and n is the number of bins of the histogram H1 and the histogram H2.

Hellinger Distance The Hellinger distance d (x ₁ , x ₂ ) is expressed by Expression (7). Here, Σ in equation (7) represents the total sum from i = 1 to i = n, and n is the number of bins of the histogram H1 and the histogram H2.

· Batacharia (Bhattacharyya) Distance Batacharia distance d (x _1, x ₂₎ can be expressed by equation (8). Here, Σ in equation (8) represents the total sum from i = 1 to i = n, and n is the number of bins of the histogram H1 and the histogram H2.

・ Histogram intersection method (Histogram Intersection)
The histogram crossover method is a method in which a small value is selected for each bin to be compared and the sum is divided by the total number, as shown in equation (9). Here, Σ in equation (9) represents the total sum from i = 1 to i = n, and n is the number of bins of the histogram H1 and the histogram H2. min (x _1i , x _2i ) indicates that the smaller of x _1i and x _2i is to be selected. In the histogram crossover method, when the histograms to be compared match, they are represented by 1 of the maximum value.

Cosine Similarity The cosine similarity is obtained by treating each coordinate of the top of the bins of the histograms H1 and H2 to be compared as a vector and calculating the COS value between the vectors, as shown in equation (10). Here, Σ in equation (10) represents the sum from i = 1 to i = n, and n is the number of bins of the histogram H1 and the histogram H2.

  The closer the cosine similarity is to one, the closer the distance between the histograms H1 and H2 to be compared. That is, when the histograms H1 and H2 to be compared coincide with each other, the cosine similarity is represented by 1 of the maximum value.

  For the distance between the histograms, for example, Earth Mover's Distance (EMD) may be used in addition to the method described above.

  In FIG. 16, the distances between the histogram H1 (1501) and the histograms 1502, 1503, and 1504 are calculated using, for example, the Manhattan distance, the Euclidean distance, and the histogram intersection method.

  Regarding the Manhattan distance and the Euclidean distance, it is evaluated that the histograms to be compared are more similar as the distance is shorter. In the histogram crossover method, the closer the histograms to be compared are, the closer to 1 the value is, and the maximum value is 1 when they match.

  From the table of FIG. 16, in comparison between the histogram H1 (1501) and the histogram H2 (1502, 1503, 1504), the histogram H1 (1501) and the histogram H2 (1503) can be used regardless of which distance calculation method is adopted. The degree of similarity is the highest.

  Thus, the similarity between images of the feature line extracted from the captured image and the three-dimensional line segment of the 3D model rotated based on the temporary corresponding segment pair is a histogram of the angle of the feature line on the captured image It is possible to calculate the distance between the angle of the projection line and the histogram of the projection line as an index.

  At this time, when the value of the distance between the histograms is equal to or more than the threshold value or less than the threshold value, the feature line extracted from the photographed image and the three-dimensional line segment of the 3D model rotated based on the temporary corresponding line segment pair. It can be estimated that the correspondence is correct. In this case, the pair estimation unit 604 determines that the temporary corresponding line segment pair has a possibility of the corresponding line segment.

  It returns to the explanation of FIG. If there is a possibility that the temporary corresponding line segment pair is a corresponding line segment, the pair estimation unit 604 searches for a pair of other corresponding line segments from the positional relationship of the line segments (step 708). As described above, in order to determine the position and orientation of the camera, at least four pairs of three-dimensional line segments and feature lines are required. Here, the pair estimation unit 604 searches for three pairs of corresponding line segments in addition to the temporary corresponding line segment pair (one pair). Specifically, when the pair estimation unit 604 fixes the temporary corresponding line segment pair determined to have the possibility of the corresponding line segment, another process is performed based on the angle and the position of the other corresponding line segments. Find line 3 pairs.

  When four or more corresponding line segment pairs are found, the viewpoint estimation unit 605 determines the position and direction of the camera viewpoint with respect to the object in the three-dimensional space from the image of the object based on the correspondence result of the corresponding line pair. The (camera position and orientation) is calculated (step 709). Then, the viewpoint estimation unit 605 calculates, for the calculated camera position and orientation, an error between a feature line and a projection line obtained by projecting a three-dimensional line segment on a captured image. Then, the viewpoint estimation unit 605 repeats steps 704 to 709, and selects and outputs a camera position and orientation that minimizes the error.

  According to such an image processing apparatus 601, it is possible to reduce the amount of calculation for correlating the image of the object with the shape information representing the shape of the object. Below, the Example of this embodiment is explained in full detail.

  FIG. 17 is a diagram showing an example of the image processing apparatus in the present embodiment (Example 1). The image processing apparatus 1701 includes a 3D model reading unit 1702, a two-dimensional line segment extraction unit 1703, an image acquisition unit 1704, a feature line detection unit 1705, a pair estimation unit 1706, a position calculation unit 1707, an error calculation unit 1708, a determination unit 1709, An output unit 1710 is included.

  The image processing device 1701 may be a portable terminal device such as a tablet, a laptop PC, or a smart device, or may be an information processing device such as a desktop PC.

  The imaging device 1740 is, for example, a camera, and captures an object to generate a captured image 1722. The image acquisition unit 1704 acquires the photographed image 1722 captured by the imaging device 1740 and stores the acquired image in the storage unit 1721. The storage unit 1721 stores the captured image 172 2D model 1723, the feature line 1725, the three-dimensional line segment 1726, the corresponding line segment pair 1727, the rough parameter 1728, the index 1729, and the parameter 1730.

  The 3D model reading unit 1702 reads the 3D model 1723 corresponding to the imaging target, and stores the 3D model 1723 in the storage unit 1721. As described above, the 3D model 1723 is an example of three-dimensional shape information, and includes vertex information of a plurality of vertices representing a three-dimensional shape of an object, and line segment information of a plurality of line segments. The vertex information includes three-dimensional coordinates of each vertex of the object, and the line segment information includes identification information indicating vertices at both ends of each line segment.

  The three-dimensional line segment extraction unit 1703 extracts the outline (three-dimensional line segment) of the object in the 3D model 1723 and stores the outline as a three-dimensional line segment 1726 in the storage unit 1721. The three-dimensional line segment extraction unit 1703 corresponds to the three-dimensional line segment extraction unit 602 described above.

  The feature line detection unit 1705 performs edge detection processing to detect a plurality of edge lines from the captured image 1722, and stores the detected edge lines in the storage unit 1721 as feature lines 1725. A feature line detection unit 1705 corresponds to the feature line extraction unit 603 described above.

  The pair estimation unit 1706 selects one three-dimensional line segment from one of the detected feature lines and one of the extracted three-dimensional line segments, estimates a corresponding line segment pair, and estimates the corresponding line segment. Other corresponding line segment pairs are also estimated based on the pairs. In the present embodiment, the pair estimation unit 1706 performs the processing of step 703 to step 708 in FIG. 7 to estimate N (where N.gtoreq.4) pairs of corresponding line segment pairs 1727 and store them in the storage unit 1721. Do. The pair estimation unit 1706 corresponds to the above-described pair estimation unit 604.

  The position calculation unit 1707 estimates an estimated value of the position and orientation of the camera using the selected corresponding line segment pair (here, the estimated value is a value before error correction, and hereinafter, will be referred to as a “rough estimated value”). calculate. That is, the position calculation unit 1707 calculates rough estimation values of the position and orientation of the imaging device 1740 when the photographed image 1722 is photographed using N pairs of corresponding line segments 1727, and calculates the rough estimation values. It is stored in the storage unit 1721 as a parameter 1728.

  At this time, the position calculation unit 1707 projects the three-dimensional line segment included in each corresponding line segment pair onto the captured image 1722 while changing the variable of the rough estimated value representing the position and orientation of the imaging device 1740 by a predetermined value. By doing this, a projection line is generated.

  An error calculation unit 1708 calculates an error between a feature line and a projection line obtained by projecting a three-dimensional line segment on a captured image, with respect to the calculated rough estimated value of the camera position and orientation. That is, the error calculation unit 1708 calculates an error representing a deviation between the position of the projection line generated by the position calculation unit 1707 and the position of the feature line included in the corresponding line segment pair. Then, the position calculation unit 1707 obtains, as a coarse parameter 1728, a value of a variable that minimizes the total sum of the errors calculated by the error calculation unit 1708.

  The position calculation unit 1707 repeats the process of calculating the coarse parameter 1728 a plurality of times while changing the selection of the N corresponding line segment pairs.

  The determination unit 1709 repeatedly executes the processing of the pair estimation unit 1706, the position calculation unit 1707, and the error calculation unit 1708 until it becomes a predetermined number of times or less than a predetermined error, and uses the camera position and posture with the smallest error as an estimated value. decide.

  That is, determination unit 1709 is included in the N corresponding line segment pairs selected by position calculation unit 1707 using the position and orientation of imaging device 1740 represented by rough parameter 1728 every time rough parameter 1728 is calculated. The three-dimensional line segment to be projected is projected onto the photographed image 1722. Thereby, the determination unit 1709 generates N projection lines. Then, the determination unit 1709 calculates the sum of the errors between the positions of the N projection lines and the positions of the N feature lines in the same manner as the error calculation unit 1708, and calculates the sum of the calculated errors as an index 1729. And stored in the storage unit 1721 as

  Next, the determination unit 1709 determines N pairs of corresponding line segments that minimize the sum of errors, based on the indices 1729 calculated using the respective rough parameters 1728. These N corresponding line segment pairs correspond to the association results. The determining unit 1709 then calculates the position and orientation of the imaging device 1740 in the three-dimensional space using the determined N corresponding line segment pairs, and stores the calculated position and orientation in the storage unit 1721 as the parameter 1730. Do.

  The position calculation unit 1707, the error calculation unit 1708, and the determination unit 1709 correspond to the viewpoint estimation unit 605 described above. The output unit 1710 outputs the parameter 1730 as a processing result.

  According to such an image processing apparatus 1701, a pair of corresponding line segments of a three-dimensional line segment detected from the 3D model 1723 and a feature line detected from the captured image 1722 is automatically performed without the user performing a selection operation. Can be generated. At this time, since the generation unit 1707 narrows down the three-dimensional line segment and the feature line used to generate the corresponding line segment pair, at least four pairs of corresponding line segment pairs are determined as compared with the case of performing round-robin calculation. The amount of computation for this is significantly reduced.

  FIG. 18 is a functional block diagram of a pair estimation unit in the present embodiment (Example 1). The pair estimation unit 1706 functions as a feature line selection unit 1801, a three-dimensional line segment selection unit 1802, an initial setting unit 1803, a rotation control unit 1804, a feature line angle histogram generation unit 1805, and a projection line angle histogram generation unit 1806. Further, the pair estimation unit 1706 functions as a histogram distance calculation unit 1807 and a temporary pair determination unit 1808.

  The feature line selection unit 1801 selects any one feature line from the plurality of feature lines detected by the feature line extraction unit 1705. The three-dimensional line segment selection unit 1802 selects any one three-dimensional line segment from the plurality of three-dimensional line segments extracted by the three-dimensional line segment extraction unit 1703.

  The initial setting unit 1803 rotates the 3D model 1723 so that the selected three-dimensional line segment can be seen in the same direction as the selected feature line, as described with reference to FIGS. 9 and 10.

The rotation control unit 1804 rotates the 3D model 1723 so that the projection direction of the selected three-dimensional line segment does not change, as described with reference to FIG. That is, the rotation control unit 1804 rotates the 3D model 1723 by a predetermined angle with the line segment L _mdl as an axis and with the normal line n between the line segment L _img and the camera viewpoint as an axis.

  The feature line angle histogram generation unit 1805 calculates a histogram H1 for the direction (angle) of the plurality of feature lines detected by the feature line extraction unit 1705 as described in step 707 of FIG. 7.

  The projection line angle histogram generation unit 1806, as described in step 707 in FIG. 7, projects the direction (the angle of the projection line of the 3D model when the 3D model 1723 formed by the plurality of The histogram H2 for) is calculated.

  The histogram distance calculation unit 1807 calculates the distance between the histogram H1 of the angle of the trace and the histogram H2 of the direction (angle) of the projection line of the 3D model as described in step 707 of FIG.

  The provisional pair determination unit 1808 determines whether there is a possibility of a correct correspondence between a combination of a three-dimensional line segment selected from the distances between the histograms and the selected feature line. If there is a possibility of a correct correspondence, the temporary pair determination unit 1808 determines the position of the projection line of another three-dimensional line segment based on the combination of the selected three-dimensional line segment and the selected feature line. The positional relationship between the relationship and the feature line is checked to determine the correctness of the temporary corresponding line segment pair.

  FIG. 19 is a flowchart showing a specific example of the image processing in the present embodiment (Example 1). First, the 3D model reading unit 1702 reads the 3D model 1723 and stores it in the storage unit 1721 (Step 1901). The three-dimensional line segment extraction unit 1703 extracts a plurality of three-dimensional line segments 1726 from the read 3D model 1723 (step 1902).

  The image acquisition unit 1704 acquires the captured image 1722 from the imaging device 1740 (step 1903). The feature line detection unit 1705 detects a plurality of feature lines 1725 from the captured image 1722 (step 1904).

  The pair estimation unit 1706 (feature line angle histogram generation unit 1805) calculates the direction (angle) of each detected feature line as described in step 707 of FIG. A histogram H1 of angles is generated (step 1905).

  The pair estimation unit 1706 executes an estimation process of the corresponding line segment pair (step 1906). The details of step 1906 will be described using FIG.

  FIG. 20 is a flowchart of detailed processing of the corresponding line segment pair estimation process (step 1906) in the present embodiment (Example 1). As shown in FIG. 9B, the pair estimation unit 1706 virtually arranges the camera at an appropriate position in a virtual space, and the 3D model 1723 at an approximate shooting distance on the line of sight.

The feature line selection unit 1801 selects one feature line L _img from the feature lines detected in step 1904 (step 2001). The three-dimensional line segment selecting unit 1802 selects one three-dimensional line segment L _mdl from the three-dimensional line segment 1726 detected in step 1902 (step 2002).

The initial setting unit 1803 sets the pair of the selected line segment L _img and the line segment L _mdl as a temporary corresponding line segment pair. As described in step 705 of FIG. 7, the initial setting unit 1803 causes the line segment L _mdl to be parallel to the line segment L _img on the projection plane, that is, the angles of the temporary corresponding line segment pair coincide with each other. , 3D model 1723 is rotated (step 2003).

As described in step 706 of FIG. 7, the rotation control unit 1804 specifies the 3D model 1723 with the line segment L _mdl as the axis and the normal line n of the plane formed by the line segment L _img and the camera viewpoint as the axis. Rotate by angle. The rotation control unit 1804 projects the 3D model 1723 on the projection plane every time the 3D model 1723 is rotated by a predetermined angle, and obtains direction vectors (angles) of all projection lines on the projection plane (step 2004).

  The projection line angle histogram generation unit 1806 generates a histogram H2 about the detected angle of each projection line in d (d: integer) degree units as described in step 707 of FIG. 7 (step 2005).

  The histogram distance calculation unit 1807 calculates the distance between the histogram H1 of the feature line angle and the histogram H2 of the projection line angle as described in step 707 of FIG. 7 (step 2006).

  The temporary pair determination unit 1808 evaluates the calculated distance in accordance with the adopted method of calculating the distance. For example, it is assumed that the adopted distance calculation method is the Manhattan distance calculation method or the Euclidean distance calculation method. At this time, if the calculated distance is equal to or less than the threshold value, the temporary pair determination unit 1808 determines that the temporary corresponding line segment pair has a possibility of a corresponding line segment pair (“YES” in step 2007). If the calculated distance is larger than the threshold value, the temporary pair determination unit 1808 determines that the temporary corresponding line segment pair has no possibility of the corresponding line segment pair (“NO” in step 2007).

  Also, for example, it is assumed that the adopted method of calculating the distance is the histogram crossing method. At this time, if the calculated distance is equal to or larger than the threshold, the temporary pair determination unit 1808 determines that the temporary corresponding line segment pair has a possibility of a corresponding line segment pair (“YES” in step 2007). If the calculated distance is less than the threshold value, the temporary pair determination unit 1808 determines that the temporary corresponding line segment pair has no possibility of the corresponding line segment pair (“NO” in step 2007).

  If it is determined that the temporary corresponding line segment pair is likely to be a corresponding line segment pair, the temporary pair determination unit 1808 estimates three other corresponding line pair pairs (step 2008). Step 2008 will be described with reference to FIG.

If it is determined that there is no possibility of a corresponding line segment pair ("NO" in step 2007), a total of four pairs of corresponding line segment pairs can not be found by the processing of step 2008 (step 2009). And the process of step 2010 is performed. That is, the temporary pair determination unit 1808 determines whether or not the processing after step 2003 has been completed for all positions when the 3D model 1723 is rotated. With all the positions when the 3D model 1723 is rotated, as described in FIG. 12, the normal line n of the plane formed by the line segment L _img and the camera viewpoint with the three-dimensional line segment L _mdl as an axis is an axis The 3D model 1723 is rotated about each axis, for example, one degree at a time, to represent all positions.

  When the process is not completed for all the positions when the 3D model 1723 is rotated (“NO” in step 2010), the rotation control unit 1804 rotates the 3D model 1723 to a position where the process is not completed ( Step 2003).

  If a total of four pairs of corresponding line segment pairs can not be found, the temporary pair determination unit 1808 repeats the processing of step 2003 to step 2010 until the processing is completed for all positions when the 3D model 1723 is rotated.

  When the processing is completed for all the positions when the 3D model 1723 is rotated (“YES” in step 2010), the temporary pair determination unit 1808 determines whether there is an unselected three-dimensional line segment (step 2011). ). If there is an unselected three-dimensional line segment (“YES” in step 2011), the three-dimensional line segment selecting unit 1802 selects one of the unselected three-dimensional line segments, and step 2002 Perform the following processing.

  If a total of four pairs of corresponding line segment pairs are not found, the temporary pair determination unit 1808 repeats the processes of step 2002 to step 2011 until the process is completed for all three-dimensional line segments.

  When processing for all three-dimensional line segments is completed when a total of four pairs of corresponding line segment pairs are not found (“NO” in step 2011), the feature line selection unit 1801 selects one of the unselected feature lines. One is selected, and the processing after step 2001 is performed.

  If a total of four pairs of corresponding line segment pairs can not be found, the temporary pair determination unit 1808 repeats the processes of steps 2001 to 2011 until the process is completed for all feature lines.

  If a total of four pairs of corresponding line segments are found (“YES” in step 2009), the pair estimation unit 1706 ends this flow.

FIG. 21 is a flowchart of a detailed process of the process (step 2008) of estimating another three pairs of corresponding line segment pairs in the present embodiment (Example 1). The temporary corresponding line segment pair set in step 2002 is formed of the feature line L _img and the three-dimensional line segment L _mdl . Here, for convenience of description of FIG. 21, the feature line L _img is referred to as a first feature line, and the projection line of the three-dimensional line segment L _mdl is referred to as a first projection line. Further, a pair of the feature line L _img and the three-dimensional line segment L _mdl , that is, a temporary corresponding line segment pair determined to have a possibility of a corresponding line segment pair is referred to as a first corresponding line segment pair.

  Among the projection lines of the 3D model 1723 obtained in step 2004 (except for the first projection line), the temporary pair determination unit 1808 performs a second projection line whose edge angle matches the feature line. It selects as a projection line of (step 2101).

  The temporary pair determination unit 1808 sets, as a second feature line, a feature line whose angle coincides with the projection line selected in step 2101 out of the feature lines detected in step 1904 (except for the first feature line). The selection is made (step 2102). At this time, the temporary pair determination unit 1808 estimates a pair of the second feature line and the three-dimensional line segment corresponding to the second projection line as a second corresponding line segment pair.

  The temporary pair determination unit 1808 performs scaling adjustment processing (step 2103). Since the Z direction of the camera has been processed using provisional values so far, the pair estimation unit 1706 adjusts the projection size of the 3D model.

  The temporary pair determination unit 1808 determines that the angle of the edge matches the feature line from among the projection lines of the 3D model 1723 obtained in step 2004 (except for the first projection line and the second projection line). The projection line existing is selected as a third projection line (step 2104).

  The temporary pair determination unit 1808 selects a feature line whose angle and position match the third projection line out of the feature lines detected in step 1904 (except for the first feature line and the second feature line). That is, it is determined whether there is a third feature line (step 2105).

  If there is no third feature line (“NO” in step 2105), the temporary pair determination unit 1808 performs the process of step 2101 and subsequent steps on the unselected projection lines. That is, the temporary pair determination unit 1808 selects a projection line whose edge angle matches the feature line as a second projection line from among the unselected projection lines (except for the first projection line). Then (step 2101), the process is repeated as described above.

  If there is a third feature line (“YES” in step 2105), the temporary pair determination unit 1808 generates a third pair of the third feature line and the three-dimensional line segment corresponding to the third projection line. Estimate as corresponding line pair.

  Then, from among the projection lines of the 3D model 1723 obtained in step 2004 (except for the first to third projection lines), the temporary pair determination unit 1808 matches the angle and position of the edge with the feature line. The selected projection line is selected as a fourth projection line (step 2106).

  The temporary pair determination unit 1808 selects a feature line whose angle and position match the fourth projection line from among the feature lines detected in step 1904 (except for the first to third feature lines), that is, the fourth feature line It is determined whether there is a feature line of (step 2107).

  If there is a fourth feature line (“YES” in step 2107), the temporary pair determination unit 1808 generates a fourth pair of the fourth feature line and the three-dimensional line segment corresponding to the fourth projection line. Estimate as corresponding line pair. In this case, the temporary pair determination unit 1808 determines that there are a total of four pairs of corresponding line segments (step 2110).

  If there is no fourth feature line (“NO” in step 2107), the temporary pair determination unit 1808 performs the process of step 2101 and subsequent steps on the unselected projection lines (“NO” in step 2108). That is, the temporary pair determination unit 1808 selects a projection line whose edge angle matches the feature line as a second projection line from among the unselected projection lines (except for the first projection line). Then (step 2101), the process is repeated as described above. The processing of steps 2101 to 2108 is performed for all projection lines (except for the first projection line).

  When the processing of step 2101 to step 2108 is completed for all projection lines (except for the first projection line) (“YES” in step 2108), the temporary pair determination unit 1808 determines a total of four pairs of corresponding lines. It is determined that a minute pair does not exist (step 2109).

  It returns to the explanation of FIG. After the corresponding line segment pair estimation process (step 1906), the position calculation unit 1707 uses the N (corresponding to N = 4 in the present embodiment) corresponding line segment pair 1727 to indicate a rough camera position and orientation (hereinafter referred to as “ The rough parameter (referred to as “coarse parameter”) is calculated (step 1907). The coarse camera position attitude indicates a camera position attitude including an error.

  FIG. 22 is a diagram showing an example of the corresponding line segment pair in the present embodiment (Example 1). In this example, a three-dimensional line segment 2211 to a three-dimensional line segment 2214 are respectively associated with the feature line 2201 to the feature line 2204 to form four corresponding line segment pairs.

  In step 1907, the position calculation unit 1707 can calculate the coarse parameter 1728 using, for example, the least squares method. In this case, the position calculation unit 1707 projects three-dimensional line segments included in each corresponding line segment pair onto the captured image 1722 while changing variables representing the position and orientation of the imaging device 1740 by predetermined values. Generate a projection line.

  The error calculation unit 1708 evaluates an error Ei (i = 1 to N) between the position of the projection line and the position of the feature line included in the corresponding line segment pair, and calculates the square error of four corresponding line segment pairs. The value of the variable that minimizes the sum E is determined as a coarse parameter 1728. The sum E of squared errors is calculated by the following equation (11).

  The error calculator 1708 can calculate the error Ei, for example, by the method shown in FIG. 23 or 24. FIG. 23 shows an example of a calculation method based on the area of the region between the projection line and the feature line. When the projection line included in the i-th corresponding line pair is a line segment 2301 and the feature line is a line segment 2302, a line segment 2303 and a line segment respectively connecting both ends of the line segment 2301 and both ends of the line segment 2302 2304 can be defined. In this case, the area Ai of the region surrounded by the line segments 2301 to 2304 can be used as the error Ei.

Ei = Ai (12)

  As the area Ai is smaller, the error Ei is smaller. When the line segment 2301 overlaps the line segment 2302, the error Ei is zero.

  FIG. 24 shows an example of a calculation method based on the distance between the projection line and the feature line. Let the lengths of the perpendicular 2401 and the perpendicular 2402 dropped onto the line segment 2301 from both ends of the line segment 2301 be Li1 and Li2, respectively. In this case, the sum of Li1 and Li2 can be used as the error Ei.

Ei = Li1 + Li2 (13)

  As Li1 and Li2 become shorter, the error Ei becomes smaller, and when the line segment 2301 overlaps the line segment 2302, the error Ei becomes zero.

  Next, the determination unit 1709 uses the position and orientation of the imaging device 1740 represented by the coarse parameter 1728 to project a three-dimensional line segment included in the N corresponding line segment pairs onto the captured image 1722 to obtain N. A projection line of a book is generated (step 1908).

  Next, the determination unit 1709 calculates an index 1729 representing the sum of errors between the positions of the N projection lines and the positions of the N feature lines (step 1909), and performs calculation of the index 1729 a predetermined number of times. Whether or not it is checked (step 1910). When the index 1729 has not been calculated a predetermined number of times (“No” in step 1910), the position calculation unit 1707 changes the selection of N corresponding line segment pairs (step 1906), and the image processing apparatus 1701 The process after 1907 is repeated.

  When the index 1729 has been calculated a predetermined number of times (“Yes” in step 1910), the determination unit 1709 selects N corresponding line segment pairs that minimize the sum of errors (step 1911). Then, the determination unit 1709 calculates a parameter (hereinafter, referred to as “parameter”) 1730 indicating the camera position and orientation based on the corresponding line segment pair (step 1912). Then, the output unit 1710 outputs the selected N corresponding line segment pairs and the parameter 1730 (step 1913).

  According to the image processing of the present embodiment, by repeating the calculation of the index 1729 while automatically changing the selection of the N corresponding line segment pairs, the N corresponding line segment pairs for which the total sum of the errors is minimized It can be asked. As a result, the working time of the selection operation by the user is reduced, the processing time is shortened, and the estimation accuracy of the position and orientation of the imaging device 1740 is improved.

  In addition, since there is no selection error due to human error, there is no increase in processing time due to reselection. Since it is possible to determine the optimum N corresponding line segment pairs without being a skilled person, it is possible to expand the types of work and the target person to which the correspondence result is applied.

  In step 1910 of FIG. 19, the image processing apparatus 1701 repeatedly performs the process when the error represented by the index 1729 is smaller than a predetermined value, instead of terminating the process repeatedly when the calculation of the index 1729 is performed a predetermined number of times. You may

  In addition, in steps 1907 and 1909, the image processing device 1701 may evaluate the similarity between the projection line and the feature line instead of the error between the projection line position and the feature line position. . As the similarity between the projection line and the feature line, for example, the similarity between two line segments described in Patent Document 3 can be used. In this case, in step 1907, a rough parameter 1728 is obtained which maximizes the sum of the similarities, and in step 1911 N corresponding line segment pairs which maximize the sum of the similarities are selected.

  By the way, even if the total sum of the errors of the N corresponding line segment pairs selected in step 911 is minimum, each projection line may represent a line segment obtained by rotating each feature line by 180 degrees.

  FIG. 25 shows an example of a line segment rotated 180 degrees. Of the projection lines and the feature lines in FIG. 25, the projection line 2512 overlaps the feature line 2502. On the other hand, the projection line 2511, the projection line 2513, and the projection line 2514 correspond to line segments obtained by rotating the feature line 2502, the feature line 2503, and the feature line 2504 by 180 degrees around the projection line 2512 as an axis. In this case, since the area Ai of equation (12) is approximately zero, the sum of errors may be minimized.

  Therefore, in order to prohibit such correspondence, the determination unit 1709 selects N corresponding line segment pairs for which the sum of errors is minimized among N corresponding line segment pairs satisfying the following conditions. May be selected.

(C11) Among the N projection lines, a projection line of a predetermined ratio is included in the captured image 1722.
(C12) Of the N projection lines, a projection line of a predetermined ratio is present in the vicinity of a predetermined position in the captured image 1722.
(C13) The ratio of the distribution range of N projection lines to the area of the captured image 1722 is equal to or more than a predetermined value.

  FIG. 26 is a diagram showing an example of a three-dimensional line segment not suitable for calculation of the coarse parameter 1728. FIG. 26 (a) shows four three-dimensional line segments parallel to one another. When four three-dimensional line segments are parallel, the error does not change even if the three-dimensional line segment is parallelly moved in the direction of arrow 2601, and it becomes difficult to fix the position of the three-dimensional line segment. is there.

  FIG. 26B shows two three-dimensional line segments existing on the same straight line. When two three-dimensional line segments exist on the same straight line, the error does not change even if the three-dimensional line segment is enlarged or reduced in the direction of arrow 2602, and it may be difficult to fix the scale. .

  Therefore, in step 906 of FIG. 9, the pair estimation unit 1706 may select N three-dimensional line segments that satisfy the following conditions to generate N corresponding line segment pairs.

(C21) At least two of the N three-dimensional line segments are not parallel.
(C22) Any two of the N three-dimensional line segments do not exist on the same straight line.

  For the same reason, the pair estimation unit 1706 may select N feature lines that satisfy the following conditions to generate N corresponding line segment pairs.

(C31) At least two feature lines out of N feature lines are not parallel.
(C32) Any two feature lines out of N feature lines do not exist on the same straight line.

  Next, the present embodiment (Example 2) will be described. In the first embodiment, in the corresponding line segment pair estimation process (step 1906), when finding the corresponding line segment pair, the feature line and the three-dimensional line segment are sequentially searched. On the other hand, in the second embodiment, the characteristic line and the three-dimensional line segment are selected so that the corresponding line segment pair can be obtained more quickly while increasing the expected value that can obtain a more appropriate corresponding line segment pair.

  The image processing apparatus of the second embodiment is the same as that of the first embodiment except that the functions of the pair estimation unit 1706 of the image processing apparatus of FIG. 17 are partially different, and the other configurations, functions, and processes are not particularly mentioned. It is. Further, in the second embodiment, configurations, functions and processes similar to those of the first embodiment are denoted by the same reference numerals, and the description thereof is omitted.

  FIG. 27 is a functional block diagram of a pair estimation unit in the present embodiment (Example 2). 27, a feature line group creation unit 2703 and a three-dimensional line segment group creation unit 2704 are added to the functional block diagram of the pair estimation unit of FIG. 18, and the feature line selection unit 1801 is replaced by the feature line selection unit 2701. Three-dimensional selection unit 1802 is replaced with three-dimensional selection unit 2702.

  The feature line group creation unit 2701 groups the feature lines detected in step 1904 of FIG. 19 in the same direction (parallel). At this time, with respect to the degree of the same direction, a constant error is tolerable. The grouped feature lines are referred to as feature line groups.

  The three-dimensional line segment group creation unit 2704 groups the three-dimensional line segments detected in step 1902 of FIG. 19 together in the same direction (parallel). The grouped three-dimensional line segments are referred to as three-dimensional line segment groups.

  The feature line selection unit 2701 selects a feature line group having a large number of feature lines in the feature line group. The feature line selection unit 2701 selects a feature line from the outside in the selected feature line group when the lengths of all feature lines included in the selected feature line group are within a predetermined range. If the length of any feature line included in the selected feature line group exceeds a predetermined range, the feature line selection unit 2701 selects the longest feature line from the selected feature line group.

  The three-dimensional selection unit 2702 selects one three-dimensional line segment group including a number of three-dimensional line segments closer in number to the number of feature lines included in the selected feature line group from among a plurality of three-dimensional line segment groups. , Select one of the three-dimensional line segment groups. The three-dimensional selection unit 2702 selects the three-dimensional line segment group including the longest three-dimensional line segment, when there are a plurality of three-dimensional line segment groups including three or more near-numbered three-dimensional line segments. Do. If there is no three-dimensional line segment group including three-dimensional line segments equal to or more than the number of feature lines, the three-dimensional selection unit 2702 selects a closer number of feature lines less than the number of feature lines included in the selected feature line group. Select 3D line segment group including 3D line segments. The three-dimensional selection unit 2702 selects the longest three-dimensional line segment from the selected three-dimensional line segment group.

  When the temporary correspondence is denied by the histogram distance, the temporary pair determination unit 1808 selects the next temporary corresponding line segment pair from other three-dimensional line segment groups. If another corresponding line segment pair is not found and the temporary correspondence is denied, the temporary pair determination unit 1808 determines the direction of the rotation angle in the vicinity of the three-dimensional line segment of the same three-dimensional line segment group. Find the corresponding segment pair from the vector distribution.

  FIG. 28 is a flowchart showing a specific example of the image processing in the present embodiment (example 2). The flowchart of FIG. 28 is obtained by adding steps 2801 and 2802 to the flowchart of FIG.

  After the processing of steps 1901 to 1905, the feature line group creating unit 2701 groups the feature lines detected in step 1904 of FIG. 19 in the same direction (parallel) (step 2801). At this time, with respect to the degree of the same direction, a constant error is tolerable.

  The three-dimensional line segment group creation unit 2704 groups the three-dimensional line segments detected in step 1902 of FIG. 19 together in the same direction (parallel) (step 2802).

  Thereafter, the processing of step 1906 and thereafter is performed. However, in the second embodiment, since step 2001 and step 2002 in the process of step 1906 are improved, these will be described below.

  FIG. 29 is a detailed flowchart of feature line selection (step 2001) in the present embodiment (Example 2). FIG. 29 is a flowchart detailing feature line selection (step 2001) of FIG.

  The feature line selection unit 2701 selects a feature line group having the largest number of feature lines among the unselected feature line groups (step 2901).

  The feature line selection unit 2701 determines whether the lengths of the feature lines included in the selected feature line group are substantially the same (step 2902). For example, when the standard deviation of the lengths of the feature lines included in the selected feature line group is calculated, and the lengths of all the feature lines fall within the predetermined range, the feature line selection unit 2701 It is determined that the lengths of the feature lines are approximately the same. For example, when the standard deviation of the lengths of the feature lines included in the selected feature line group is calculated, and the lengths of all the feature lines included in the selected feature line group do not fall within the predetermined range, The feature line selection unit 2701 determines that the lengths of all the feature lines are not equal.

  If it is determined in step 2902 that the lengths of all the feature lines are not equal ("NO" in step 2902), the feature line selection unit 2701 selects one of the unselected feature lines in the selected feature line group. The longest feature line is selected (step 2903). The reason for selecting from longer feature lines is that the longer the line segment, the higher the angle accuracy of that line segment.

  If it is determined in step 2902 that the lengths of all the feature lines are substantially the same (“YES” in step 2902), the feature line selection unit 2701 selects one of the unselected feature lines in the selected feature line group. , The outermost feature line is selected as a first feature line (step 2904). The reason for selecting from the outer line segments is that it is expected that the relative distance to the other selected line segment will be long, and as a result, it can be expected that a large deviation will occur in the case of incorrect correspondence.

  Here, as a method of selecting from the outer feature lines, for example, there is a method of selecting from the outer peripheral line first. Here, whether or not it is an outer peripheral line may be determined, for example, in a group of a plurality of parallel line segments, if there is a feature line on one side of a certain feature line and no feature line on the other side. it can.

  FIG. 30 is a diagram for describing a method of selecting from outer feature lines in the present embodiment (Example 2). In FIG. 30, the feature lines 3001, 3002, and 3003 are approximately parallel, so they are the same group. At this time, the feature line 3002 does not correspond to the outer peripheral line because the feature lines 3001 and 3003 are present on both sides thereof. On the other hand, there is a feature line 3102 on the left side of the feature line 3003, but there is no feature line on the left side of the feature line 3003, so the feature line 3103 corresponds to an outer peripheral line.

  Further, as another method of selecting from the outer characteristic lines, there is, for example, a method of selecting from a plurality of outer peripheral lines having the same length or selecting from line segments that are not outer peripheral lines. For example, the feature line selection unit 2701 may select from feature lines that are far from the center of gravity of all the line segments. Also, for example, the maximum distance between any point on the straight line extending the line segment and any point on the other line segment may be selected from a long feature line.

  FIG. 31 is a flowchart of detailed processing of three-dimensional line segment selection (step 2002) in the present embodiment (Example 2). FIG. 31 is a flowchart detailing feature line selection (step 2002) of FIG.

  The three-dimensional line segment selection unit 2702 determines whether there are three-dimensional line segment groups having the number of feature lines in the selected feature line group or more among the unselected three-dimensional line segment groups (step 3101). ).

  When there is a three-dimensional line segment group having the number (M1) or more of the feature lines in the selected feature line group among the unselected three-dimensional line segment groups (“YES” in step 3101) The minute selecting unit 2702 performs the following. That is, the three-dimensional line segment selection unit 2702 selects a three-dimensional line segment group having the number (M2) closest to M1 or more and M1 (step 3103). As a result of selection, when one group of three-dimensional line segment groups having M1 or more and the number (M2) closest to M1 is selected (“YES” in step 3104), the three-dimensional line segment selection unit 2702 proceeds to step 3106. Do the processing.

  As a result of selection, when a plurality of three-dimensional line segment groups having the number (M2) closest to M1 and not less than M1 are selected (“NO” in step 3104), the three-dimensional line segment selection unit 2702 A three-dimensional line segment group having dimensional line segments is selected (step 3105).

  When there is no three-dimensional line segment group having the number (M1) or more of the feature lines in the selected feature line group among the unselected three-dimensional line segment groups ("NO" in step 3101), the three-dimensional line The minute selecting unit 2702 performs the following. That is, the three-dimensional line segment selecting unit 2702 selects the three-dimensional line segment group having the closest number of feature lines in the selected feature line group less than the number (M1) among the unselected three-dimensional line segment groups. (Step 3102).

  If “YES” in step 3104, or if step 3102 ends, or if step 3105 ends, the three-dimensional line segment selection unit 2702 selects one of the unselected three-dimensional line segments in the selected three-dimensional line segment group. The longest three-dimensional line segment is selected (step 3106).

  The reason for selecting a three-dimensional line segment group in which M1 ≦ M2 in step 3103 is because the number of feature lines is often equal to or less than the number of corresponding three-dimensional line segments. If there is an erroneous grouping, it is reversed.

  The reason for selecting from the long three-dimensional line segments in step 3105 is that since the feature lines are selected from the long ones, it is expected that the three-dimensional line segments are also long. If the three-dimensional line segment group is not found, the reason for searching from M1> M2 is also because there is a possibility of erroneous grouping.

  The reason for selecting from long 3D line segment groups is that feature lines are selected from long ones, so if you choose from long 3D line segments, the expected value for finding the correct correspondence is higher. .

  When the three-dimensional line segment of the group selected in steps 3102, 3103, 3105 is the distribution of the direction vector and the temporary corresponding line segment pair is negated at all angles in the three-dimensional line segment (Step 2007 in FIG. NO "), the temporary pair determination unit 1808 performs the following. That is, the temporary pair determination unit 1808 selects the next corresponding line segment pair from other three-dimensional line segment groups. The selection of the next corresponding line segment pair from other three-dimensional line segment groups is that the line segments of the three-dimensional line segment group are parallel, so the distribution of the slope of the line segment rotated about any of them is the same It is almost the same as the inclination of the line segment rotated around the other line segments in the group. Therefore, if the selected line segment is rejected in the distribution, the other line segments in the same group are also highly likely to be rejected.

When the corresponding line segment pair is not found and the corresponding line segment pair is negated (“NO” in step 2009 of FIG. 20), the temporary pair determination unit 1808 determines that the three-dimensional line segment in the same three-dimensional line segment group is adjacent The corresponding line segment pair is searched from the direction vector distribution of the rotation angle of. Here, the rotation angle in the vicinity indicates the vicinity of the rotation angle in the case of “YES” in step 2007.
Direction vector distributions of nearby rotation angles are searched for in three-dimensional line segments in the same three-dimensional line segment group. Here, since the line segments of the three-dimensional line segment group are parallel, the distribution of the inclination of the line segment rotated about one of them is rotated about the other line segment in the same three-dimensional line segment group It is almost the same as the slope of the line segment. Therefore, in order to find the corresponding line segment pair, it is sufficient to search the angle in the vicinity.

  FIG. 32 is a diagram for describing a method of selecting a corresponding line segment in the present embodiment (Example 2). FIG. 32A shows an example of the feature line detected from the photographed image 1722. FIG. 32 (b) shows three-dimensional line segments detected in the 3D model.

  (S11) In FIG. 32A, the group of feature lines 3201 to 3203 and the group of feature lines 3204 to 3206 each have three feature lines.

  (S12) The feature line selection unit 2701 selects a group of feature lines 3201 to 3203 having the longest feature line among the two groups.

  (S13) The three-dimensional line segment selection unit 2702 selects a three-dimensional line segment from the four three-dimensional line segments 3211 to 3214 because there is no group having three three-dimensional line segments.

  (S14) The longest three-dimensional line segment 3211 can not be selected due to the positional relationship with other line segments in the other corresponding line segment pair estimation process (step 2008).

  (S15) It is possible to select a second longest three-dimensional line segment 3212 from three-dimensional line segments in the same group.

  Next, the feature line selection unit 2701 selects the groups 3204 to 3206 in S12.

  (S13 ') Since there is no group having three three-dimensional line segments, the three-dimensional line segment selection unit 2702 selects three-dimensional line segments from the four three-dimensional line segments 3211 to 3214.

  (S14 ') The longest three-dimensional line segment 3211 among the selected three-dimensional line segments is all angles obtained by rotating the 3D model, and there is no possibility of corresponding line segment pair ("NO" at step 1907 in FIG. 21). ). Therefore, it is assumed that the three-dimensional line segment selection unit 2702 selects from the five three-dimensional line segments 3215 to 3219. In this case, "YES" is obtained in step 2007 in FIG. 20, or the corresponding line segment pair is not found, and the corresponding line segment pair is negated ("NO" in step 2009).

  (S15 ') In this case, the temporary pair determination unit 1808 searches for the corresponding line segment pair based on the direction vector distribution of the rotation angle in the vicinity of the three-dimensional line segments 3215 to 3219 in the same three-dimensional line segment group.

  According to the second embodiment, it is possible to increase the expected value for obtaining the correct corresponding line segment pair when selecting a three-dimensional line segment, and to speed up the process of searching for the corresponding line segment pair.

  The configuration of the image processing apparatus 1701 in FIG. 17 is merely an example, and some components may be omitted or changed depending on the application or conditions of the image processing apparatus 1701. For example, when the process of detecting the feature line 1725 from the captured image 1722 is performed by an apparatus outside the image processing apparatus 1701, the feature line detection unit 1705 in FIG. 17 can be omitted.

  When the process of extracting the three-dimensional line segment 1726 from the 3D model 1723 is performed by an external device, the three-dimensional line segment extraction unit 1703 can be omitted.

  FIG. 33 is a view showing a configuration example of an information processing apparatus (computer) used as the image processing apparatus 1701 in the present embodiment (Example 1, Example 2). The information processing device 3300 includes a central processing unit (CPU) 3301, a memory 3302, an input device 3303, an output device 3304, an auxiliary storage device 3305, a medium drive device 3306, and a network connection device 3307. These components are connected to one another by a bus 3308. The imaging device 1740 of FIG. 17 may be connected to the bus 3308.

  The memory 3302 is, for example, a semiconductor memory such as a read only memory (ROM), a random access memory (RAM), or a flash memory, and stores programs and data used for image processing. The memory 3302 can be used as the storage unit 1721 in FIG.

  The CPU 3301 (processor) operates as a pair estimation unit 1706 and a determination unit 1709, for example, by executing a program using the memory 3302. Furthermore, the CPU 3301 also operates as a 3D model reading unit 1702, a two-dimensional line segment extraction unit 1703, an image acquisition unit 1704, a feature line detection unit 1705, a position calculation unit 1707, and an error calculation unit 1708 in FIG.

  The input device 3303 is, for example, a keyboard, a pointing device, etc., and is used to input an instruction or information from an operator or a user. The output device 3304 is, for example, a display device, a printer, a speaker, or the like, and is used to inquire or instruct an operator or a user, and to output a processing result. The processing result may be the N corresponding line segment pairs determined by the determination unit 613. The output device 3304 can be used as the output unit 1710 in FIG.

  The auxiliary storage device 3305 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device or the like. The auxiliary storage device 3305 may be a hard disk drive. The information processing apparatus can store programs and data in the auxiliary storage device 3305, load them into the memory 3302, and use them. The auxiliary storage device 3305 can be used as the storage unit 1721 of FIG.

  A medium drive 3306 drives a portable recording medium 3309 and accesses the recorded contents. The portable recording medium 3309 is a memory device, a flexible disk, an optical disk, a magneto-optical disk or the like. The portable recording medium 3309 may be a Compact Disk Read Only Memory (CD-ROM), a Digital Versatile Disk (DVD), a Universal Serial Bus (USB) memory, or the like. An operator or a user can store programs and data in this portable recording medium 3309, load them into the memory 3302, and use them.

  Thus, the computer readable recording medium for storing programs and data used for image processing is physically (non-temporary) such as the memory 3302, the auxiliary storage device 3305, or the portable recording medium 3309. It is a recording medium.

  The network connection device 3307 is a communication interface that is connected to a communication network such as a Local Area Network, a Wide Area Network, etc., and performs data conversion involved in communication. The information processing device can receive programs and data from an external device via the network connection device 3307, load them into the memory 3302, and use them. The network connection device 3307 can be used as the output unit 1710 in FIG.

  Note that the information processing apparatus need not include all the components shown in FIG. 33, and some components may be omitted depending on the application or conditions. For example, when the portable recording medium 3309 or the communication network is not used, the medium driving device 3306 or the network connection device 3307 may be omitted.

  While the disclosed embodiments and their advantages have been described in detail, those skilled in the art can make various changes, additions, and omissions without departing from the scope of the present invention as set forth in the claims. I will.

Further, the following appendices will be disclosed regarding the embodiment described above.
(Supplementary Note 1)
A three-dimensional line segment extraction unit for extracting a plurality of three-dimensional line segments forming the three-dimensional shape from three-dimensional shape information indicating a three-dimensional shape of an object;
A feature line extraction unit that extracts a plurality of feature lines indicating features of the shape of the target object from a captured image obtained by capturing the target object with the imaging device;
A first distribution showing a distribution of the angles of the plurality of feature lines, and a second distribution showing a distribution of angles of a plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection surface Estimating a pair of a first feature line of the plurality of feature lines and a first three-dimensional line segment of the plurality of three-dimensional line segments based on the distance between A pair estimation unit that estimates at least three pairs of the feature line and the three-dimensional line segment based on the estimated pair of pairs;
A viewpoint estimation unit that estimates the position and direction of the imaging device with respect to the object in a three-dimensional space from the captured image based on the estimated at least four pairs;
An image processing apparatus comprising:
(Supplementary Note 2)
The pair estimation unit selects the first feature line from the extracted feature lines, and selects the first three-dimensional line segment from the extracted three-dimensional line segments.
In the virtual space, the three-dimensional shape is rotated to match the first feature line with the first three-dimensional line segment;
The three-dimensional shape in which the first feature line and the first three-dimensional line segment coincide with each other includes the first three-dimensional line segment, the first feature line, and the viewpoint of the imaging device. The three-dimensional shape is rotated by a predetermined angle with the normal to the plane being made as the rotation axis,
The second distribution is generated for the angles of the plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection plane each time it is rotated, and each of the plurality of generated second distributions The image processing apparatus according to claim 1, wherein the pair of pairs is estimated in accordance with a distance between the first distribution and the first distribution.
(Supplementary Note 3)
When the pair estimation unit estimates a first pair that is the pair, the plurality of projections other than the first projection line of the first three-dimensional line segment that forms the first pair Acquiring from the line a second projection line whose angle coincides with a second feature line other than the first feature line forming the pair, and a second corresponding to the second projection line Select two three-dimensional line segments and the second feature line, and estimate the second three-dimensional line segment and the second feature line as a second pair,
The positions and angles of the plurality of projection lines other than the first projection line and the second projection line coincide with the angle of the third feature line other than the first feature line and the second feature line. Acquiring a third projection line, selecting a third three-dimensional line segment corresponding to the third projection line and the third feature line, and selecting the third three-dimensional line segment and the third line Estimate the feature lines of the third pair,
From the plurality of projection lines other than the first projection line, the second projection line and the third projection line, other than the first feature line, the second feature line and the third feature line Acquiring a fourth projection line whose angle and position are coincident with the fourth feature line of the fourth feature line, and selecting a fourth three-dimensional line segment corresponding to the fourth projection line and the fourth feature line; The image processing apparatus according to Additional remark 1 or 2, wherein the fourth three-dimensional line segment and the fourth feature line are estimated as a fourth pair.
(Supplementary Note 4)
The pair estimation unit groups the plurality of feature lines extracted from the captured image in the same direction to form a plurality of feature line groups, and groups the plurality of three-dimensional line segments in the same direction Form a 3D line segment group of
Based on at least one condition among the number of feature lines included in each feature line group, the length of the feature lines included in each feature line group, and the position of feature lines in each feature line group , Select the first feature line,
The first one is based on at least one condition among the number of three-dimensional line segments included in each three-dimensional line segment group and the length of the three-dimensional line segment included in each three-dimensional line segment group. Select a 3D line segment of
The image processing apparatus according to any one of appendices 1 to 3, characterized in that:
(Supplementary Note 5)
The pair estimation unit groups the plurality of feature lines extracted from the captured image in the same direction to form a plurality of feature line groups, and groups the plurality of three-dimensional line segments in the same direction Form a 3D line segment group of
From the plurality of feature line groups, select the feature line group containing the most feature lines,
If lengths of all feature lines included in the selected feature line group are within a predetermined range, a feature line corresponding to an outer peripheral line in the selected feature line group is selected as the first feature line If the length of any feature line included in the selected feature line group exceeds the predetermined range, the longest feature line is selected as the first feature line from the selected feature line group,
In the case where there is one three-dimensional line segment group including a number of three-dimensional line segments closer in number to the number of the feature lines included in the selected feature line group from the plurality of three-dimensional line segment groups, Three three-dimensional line segments including the longest three-dimensional line segment when there are a plurality of three-dimensional line segments including a plurality of three-dimensional line segments having a number equal to or greater than the number of feature lines. When a minute group is selected, and there is no three-dimensional line segment group including three-dimensional line segments equal to or more than the number of feature lines, the number is closer than the number of feature lines included in the selected feature line group. Select a 3D line segment group that contains 3D line segments of
The image processing apparatus according to any one of Appendices 1 to 3, wherein a longest three-dimensional line segment is selected as the first three-dimensional line segment from the selected three-dimensional line segment group.
(Supplementary Note 6)
On the computer
Extracting a plurality of three-dimensional line segments forming the three-dimensional shape from the three-dimensional shape information indicating the three-dimensional shape of the object;
Extracting a plurality of feature lines indicating features of the shape of the object from a photographed image obtained by photographing the object by the imaging device;
A first distribution showing a distribution of the angles of the plurality of feature lines, and a second distribution showing a distribution of angles of a plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection surface Estimating a pair of a first feature line of the plurality of feature lines and a first three-dimensional line segment of the plurality of three-dimensional line segments based on the distance between At least three pairs of the feature line and the three-dimensional line segment are estimated based on the estimated pair of pairs,
An image processing program for executing a process of estimating the position and direction of the imaging device with respect to the object in a three-dimensional space from the captured image based on the estimated at least four pairs.
(Appendix 7)
When estimating the pair of pairs,
The first feature line is selected from the plurality of extracted feature lines, and the first three-dimensional line segment is selected from the plurality of extracted three-dimensional line segments;
In the virtual space, the three-dimensional shape is rotated to match the first feature line with the first three-dimensional line segment;
The three-dimensional shape in which the first feature line and the first three-dimensional line segment coincide with each other includes the first three-dimensional line segment, the first feature line, and the viewpoint of the imaging device. The three-dimensional shape is rotated by a predetermined angle with the normal to the plane being made as the rotation axis,
The second distribution is generated for the angles of the plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection plane each time it is rotated, and each of the plurality of generated second distributions The image processing program according to supplementary note 6, wherein the pair of pairs is estimated according to a distance between the first distribution and the first distribution.
(Supplementary Note 8)
When the first pair that is the pair is estimated, the pair is extracted from the plurality of projection lines other than the first projection line of the first three-dimensional line segment forming the first pair. Acquiring a second projection line whose angle coincides with a second feature line other than the first feature line forming the pair of lines; and a second three-dimensional line segment corresponding to the second projection line And the second feature line, and estimating the second three-dimensional line segment and the second feature line as a second pair,
The positions and angles of the plurality of projection lines other than the first projection line and the second projection line coincide with the angle of the third feature line other than the first feature line and the second feature line. Acquiring a third projection line, selecting a third three-dimensional line segment corresponding to the third projection line and the third feature line, and selecting the third three-dimensional line segment and the third line Estimate the feature lines of the third pair,
From the plurality of projection lines other than the first projection line, the second projection line and the third projection line, other than the first feature line, the second feature line and the third feature line Acquiring a fourth projection line whose angle and position are coincident with the fourth feature line of the fourth feature line, and selecting a fourth three-dimensional line segment corresponding to the fourth projection line and the fourth feature line; The image processing program according to appendix 6 or 7, wherein the fourth three-dimensional line segment and the fourth feature line are estimated as a fourth pair.
(Appendix 9)
When estimating the pair of pairs,
The plurality of feature lines extracted from the captured image are grouped in the same direction to form a plurality of feature line groups, and the plurality of three-dimensional line segments are grouped in the same direction to form a plurality of three-dimensional line groups Form
Based on at least one condition among the number of feature lines included in each feature line group, the length of the feature lines included in each feature line group, and the position of feature lines in each feature line group , Select the first feature line,
The first one is based on at least one condition among the number of three-dimensional line segments included in each three-dimensional line segment group and the length of the three-dimensional line segment included in each three-dimensional line segment group. Select a 3D line segment of
The image processing program according to any one of appendices 6 to 8, characterized in that
(Supplementary Note 10)
When estimating the pair of pairs,
The plurality of feature lines extracted from the captured image are grouped in the same direction to form a plurality of feature line groups, and the plurality of three-dimensional line segments are grouped in the same direction to form a plurality of three-dimensional line groups Form
From the plurality of feature line groups, select the feature line group containing the most feature lines,
If lengths of all feature lines included in the selected feature line group are within a predetermined range, a feature line corresponding to an outer peripheral line in the selected feature line group is selected as the first feature line If the length of any feature line included in the selected feature line group exceeds the predetermined range, the longest feature line is selected as the first feature line from the selected feature line group,
In the case where there is one three-dimensional line segment group including a number of three-dimensional line segments closer in number to the number of the feature lines included in the selected feature line group from the plurality of three-dimensional line segment groups, Three three-dimensional line segments including the longest three-dimensional line segment when there are a plurality of three-dimensional line segments including a plurality of three-dimensional line segments having a number equal to or greater than the number of feature lines. When a minute group is selected, and there is no three-dimensional line segment group including three-dimensional line segments equal to or more than the number of feature lines, the number is closer than the number of feature lines included in the selected feature line group. Select a 3D line segment group that contains 3D line segments of
The image processing program according to any one of appendices 6 to 8, wherein a longest three-dimensional line segment is selected as the first three-dimensional line segment from the selected three-dimensional line segment group.
(Supplementary Note 11)
The computer is
Extracting a plurality of three-dimensional line segments forming the three-dimensional shape from the three-dimensional shape information indicating the three-dimensional shape of the object;
Extracting a plurality of feature lines indicating features of the shape of the object from a photographed image obtained by photographing the object by the imaging device;
A first distribution showing a distribution of the angles of the plurality of feature lines, and a second distribution showing a distribution of angles of a plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection surface Estimating a pair of a first feature line of the plurality of feature lines and a first three-dimensional line segment of the plurality of three-dimensional line segments based on the distance between At least three pairs of the feature line and the three-dimensional line segment are estimated based on the estimated pair of pairs,
An image processing method comprising: estimating a position and a direction of the imaging device with respect to the object in a three-dimensional space from the captured image based on the estimated at least four pairs.
(Supplementary Note 12)
When estimating the pair of pairs,
The first feature line is selected from the plurality of extracted feature lines, and the first three-dimensional line segment is selected from the plurality of extracted three-dimensional line segments;
In the virtual space, the three-dimensional shape is rotated to match the first feature line with the first three-dimensional line segment;
The three-dimensional shape in which the first feature line and the first three-dimensional line segment coincide with each other includes the first three-dimensional line segment, the first feature line, and the viewpoint of the imaging device. The three-dimensional shape is rotated by a predetermined angle with the normal to the plane being made as the rotation axis,
The second distribution is generated for the angles of the plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection plane each time it is rotated, and each of the plurality of generated second distributions The image processing method according to appendix 11, wherein the pair of pairs is estimated according to a distance between the first distribution and the first distribution.
(Supplementary Note 13)
When the first pair that is the pair is estimated, the pair is extracted from the plurality of projection lines other than the first projection line of the first three-dimensional line segment forming the first pair. Acquiring a second projection line whose angle coincides with a second feature line other than the first feature line forming the pair of lines; and a second three-dimensional line segment corresponding to the second projection line And the second feature line, and estimating the second three-dimensional line segment and the second feature line as a second pair,
The positions and angles of the plurality of projection lines other than the first projection line and the second projection line coincide with the angle of the third feature line other than the first feature line and the second feature line. Acquiring a third projection line, selecting a third three-dimensional line segment corresponding to the third projection line and the third feature line, and selecting the third three-dimensional line segment and the third line Estimate the feature lines of the third pair,
From the plurality of projection lines other than the first projection line, the second projection line and the third projection line, other than the first feature line, the second feature line and the third feature line Acquiring a fourth projection line whose angle and position are coincident with the fourth feature line of the fourth feature line, and selecting a fourth three-dimensional line segment corresponding to the fourth projection line and the fourth feature line; The image processing method according to appendix 11 or 12, wherein the fourth three-dimensional line segment and the fourth feature line are estimated as a fourth pair.
(Supplementary Note 14)
When estimating the pair of pairs,
The plurality of feature lines extracted from the captured image are grouped in the same direction to form a plurality of feature line groups, and the plurality of three-dimensional line segments are grouped in the same direction to form a plurality of three-dimensional line groups Form
Based on at least one condition among the number of feature lines included in each feature line group, the length of the feature lines included in each feature line group, and the position of feature lines in each feature line group , Select the first feature line,
The first one is based on at least one condition among the number of three-dimensional line segments included in each three-dimensional line segment group and the length of the three-dimensional line segment included in each three-dimensional line segment group. Select a 3D line segment of
The image processing method according to any one of appendices 11 to 13, characterized in that
(Supplementary Note 15)
When estimating the pair of pairs,
The plurality of feature lines extracted from the captured image are grouped in the same direction to form a plurality of feature line groups, and the plurality of three-dimensional line segments are grouped in the same direction to form a plurality of three-dimensional line groups Form
From the plurality of feature line groups, select the feature line group containing the most feature lines,
If lengths of all feature lines included in the selected feature line group are within a predetermined range, a feature line corresponding to an outer peripheral line in the selected feature line group is selected as the first feature line If the length of any feature line included in the selected feature line group exceeds the predetermined range, the longest feature line is selected as the first feature line from the selected feature line group,
In the case where there is one three-dimensional line segment group including a number of three-dimensional line segments closer in number to the number of the feature lines included in the selected feature line group from the plurality of three-dimensional line segment groups, Three three-dimensional line segments including the longest three-dimensional line segment when there are a plurality of three-dimensional line segments including a plurality of three-dimensional line segments having a number equal to or greater than the number of feature lines. When a minute group is selected, and there is no three-dimensional line segment group including three-dimensional line segments equal to or more than the number of feature lines, the number is closer than the number of feature lines included in the selected feature line group. Select a 3D line segment group that contains 3D line segments of
The image processing method according to any one of Appendices 11 to 13, wherein a longest three-dimensional line segment is selected as the first three-dimensional line segment from the selected three-dimensional line segment group.

601 image processing apparatus 602 three-dimensional line segment extraction unit 603 feature line extraction unit 604 pair estimation unit 605 viewpoint estimation unit 1701 image processing apparatus 1702 3D model reading unit 1703 three-dimensional line segment extraction unit 1704 image acquisition unit 1705 feature line detection unit 1706 Pair estimation unit 1707 Position calculation unit 1708 Error calculation unit 1709 Determination unit 1710 Output unit 1721 Storage unit 1722 Captured image 1723 3D model 1725 Feature line 1726 3D line segment 1727 Corresponding line segment pair 1728 Coarse parameter 1729 Index 1730 Parameter 1801 Feature line selection Part 1802 3D line segment selection part 1803 Initial setting part 1804 Rotation control part 1805 Feature line angle histogram generation part 1806 Projection line angle histogram generation part 1807 Histogram distance calculation part 1808 Temporary pair Tough 2701 feature line selector 2702 3-dimensional selector 2703 feature line group creation unit 2704 three-dimensional line segment group creating unit

Claims (7)

A three-dimensional line segment extraction unit for extracting a plurality of three-dimensional line segments forming the three-dimensional shape from three-dimensional shape information indicating a three-dimensional shape of an object;
A feature line extraction unit that extracts a plurality of feature lines indicating features of the shape of the target object from a captured image obtained by capturing the target object with the imaging device;
A first distribution showing a distribution of the angles of the plurality of feature lines, and a second distribution showing a distribution of angles of a plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection surface Estimating a pair of a first feature line of the plurality of feature lines and a first three-dimensional line segment of the plurality of three-dimensional line segments based on the distance between A pair estimation unit that estimates at least three pairs of the feature line and the three-dimensional line segment based on the estimated pair of pairs;
A viewpoint estimation unit that estimates the position and direction of the imaging device with respect to the object in a three-dimensional space from the captured image based on the estimated at least four pairs;
An image processing apparatus comprising: The pair estimation unit selects the first feature line from the extracted feature lines, and selects the first three-dimensional line segment from the extracted three-dimensional line segments.
In the virtual space, the three-dimensional shape is rotated to match the first feature line with the first three-dimensional line segment;
The three-dimensional shape in which the first feature line and the first three-dimensional line segment coincide with each other includes the first three-dimensional line segment, the first feature line, and the viewpoint of the imaging device. The three-dimensional shape is rotated by a predetermined angle with the normal to the plane being made as the rotation axis,
The second distribution is generated for the angles of the plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection plane each time it is rotated, and each of the plurality of generated second distributions The image processing apparatus according to claim 1, wherein the pair of pairs is estimated according to a distance between the first distribution and the first distribution. When the pair estimation unit estimates a first pair that is the pair, the plurality of projections other than the first projection line of the first three-dimensional line segment that forms the first pair Acquiring from the line a second projection line whose angle coincides with a second feature line other than the first feature line forming the pair, and a second corresponding to the second projection line Select two three-dimensional line segments and the second feature line, and estimate the second three-dimensional line segment and the second feature line as a second pair,
The positions and angles of the plurality of projection lines other than the first projection line and the second projection line coincide with the angle of the third feature line other than the first feature line and the second feature line. Acquiring a third projection line, selecting a third three-dimensional line segment corresponding to the third projection line and the third feature line, and selecting the third three-dimensional line segment and the third line Estimate the feature lines of the third pair,
From the plurality of projection lines other than the first projection line, the second projection line and the third projection line, other than the first feature line, the second feature line and the third feature line Acquiring a fourth projection line whose angle and position are coincident with the fourth feature line of the fourth feature line, and selecting a fourth three-dimensional line segment corresponding to the fourth projection line and the fourth feature line; The image processing apparatus according to claim 1, wherein the fourth three-dimensional line segment and the fourth feature line are estimated as a fourth pair. The pair estimation unit groups the plurality of feature lines extracted from the captured image in the same direction to form a plurality of feature line groups, and groups the plurality of three-dimensional line segments in the same direction Form a 3D line segment group of
Based on at least one condition among the number of feature lines included in each feature line group, the length of the feature lines included in each feature line group, and the position of feature lines in each feature line group , Select the first feature line,
The first one is based on at least one condition among the number of three-dimensional line segments included in each three-dimensional line segment group and the length of the three-dimensional line segment included in each three-dimensional line segment group. Select a 3D line segment of
The image processing apparatus according to any one of claims 1 to 3, characterized in that: The pair estimation unit groups the plurality of feature lines extracted from the captured image in the same direction to form a plurality of feature line groups, and groups the plurality of three-dimensional line segments in the same direction Form a 3D line segment group of
From the plurality of feature line groups, select the feature line group containing the most feature lines,
If lengths of all feature lines included in the selected feature line group are within a predetermined range, a feature line corresponding to an outer peripheral line in the selected feature line group is selected as the first feature line If the length of any feature line included in the selected feature line group exceeds the predetermined range, the longest feature line is selected as the first feature line from the selected feature line group,
In the case where there is one three-dimensional line segment group including a number of three-dimensional line segments closer in number to the number of the feature lines included in the selected feature line group from the plurality of three-dimensional line segment groups, Three three-dimensional line segments including the longest three-dimensional line segment when there are a plurality of three-dimensional line segments including a plurality of three-dimensional line segments having a number equal to or greater than the number of feature lines. When a minute group is selected, and there is no three-dimensional line segment group including three-dimensional line segments equal to or more than the number of feature lines, the number is closer than the number of feature lines included in the selected feature line group. Select a 3D line segment group that contains 3D line segments of
The image processing apparatus according to any one of claims 1 to 3, wherein a longest three-dimensional line segment is selected as the first three-dimensional line segment from the selected three-dimensional line segment group. On the computer
Extracting a plurality of three-dimensional line segments forming the three-dimensional shape from the three-dimensional shape information indicating the three-dimensional shape of the object;
Extracting a plurality of feature lines indicating features of the shape of the object from a photographed image obtained by photographing the object by the imaging device;
A first distribution showing a distribution of the angles of the plurality of feature lines, and a second distribution showing a distribution of angles of a plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection surface Estimating a pair of a first feature line of the plurality of feature lines and a first three-dimensional line segment of the plurality of three-dimensional line segments based on the distance between At least three pairs of the feature line and the three-dimensional line segment are estimated based on the estimated pair of pairs,
An image processing program for executing a process of estimating the position and direction of the imaging device with respect to the object in a three-dimensional space from the captured image based on the estimated at least four pairs. The computer is
Extracting a plurality of three-dimensional line segments forming the three-dimensional shape from the three-dimensional shape information indicating the three-dimensional shape of the object;
Extracting a plurality of feature lines indicating features of the shape of the object from a photographed image obtained by photographing the object by the imaging device;
A first distribution showing a distribution of the angles of the plurality of feature lines, and a second distribution showing a distribution of angles of a plurality of projection lines obtained by projecting the plurality of three-dimensional line segments onto the projection surface Estimating a pair of a first feature line of the plurality of feature lines and a first three-dimensional line segment of the plurality of three-dimensional line segments based on the distance between At least three pairs of the feature line and the three-dimensional line segment are estimated based on the estimated pair of pairs,
An image processing method comprising: estimating a position and a direction of the imaging device with respect to the object in a three-dimensional space from the captured image based on the estimated at least four pairs.