WO2021075314A1 - Image processing device, image processing method, and computer-readable recording medium - Google Patents

Abstract

An image processing device 10 is provided with: a feature point extraction unit 11 for extracting combinations of feature points from images of a target object; a matrix calculation unit 12 for, when at least two corresponding line segments or points are designated, calculating a numerical matrix indicating a geometric relationship between the feature points; an unnecessary feature point specification unit 13 for specifying, from the numerical matrix, a combination of feature points the geometric relationship of which is contradicted; a three-dimensional coordinate calculation unit 14 for, by using a camera matrix of feature points calculated except for the specified combination, calculating three-dimensional coordinates corresponding to the feature points; a suitability determination unit 15 for applying the camera matrix to the three-dimensional coordinates for each of the combinations of feature points, calculating two-dimensional coordinates obtained by projecting the three-dimensional coordinates to an image of an extraction source of the feature points, and determining whether or not the feature points are suitable from comparison between the two-dimensional coordinates and the original two-dimensional coordinates of the feature points; and a three-dimensional shape construction unit 16 for constructing a three-dimensional shape of the target object by using the three-dimensional coordinates corresponding to the suitable feature points.

Description

Image processing equipment, image processing methods, and computer-readable recording media

The present invention relates to an image processing apparatus and an image processing method for enabling the construction of a three-dimensional shape from a plurality of images, and further, a computer-readable recording medium in which a program for realizing these is recorded. Regarding.

In recent years, attention has been focused on the technology for constructing the three-dimensional shape of the object shown in the image. SfM (Structure from Motion) is known as a typical example of such a technique. In SfM, a specific target is photographed a plurality of times while changing the viewpoint of the camera, and the three-dimensional shape of the specific target is reconstructed from the obtained plurality of images.

Specifically, first, in SfM, the feature amount (for example, SIFT feature amount, SURF feature amount) is calculated for each image, and robustness feature points that are resistant to image enlargement / reduction, rotation, and illuminance change are extracted. Will be done. Next, matching of the extracted feature points is executed between the images, and a pair of matching feature points is extracted. Next, for example, Robust Estimation calculates the geometrical relationship of feature point pairs and excludes erroneous feature point pairs.

Then, for each of several feature point pairs, a Fundamental matrix is calculated based on these geometric relationships, and for each feature point pair so that the difference between each calculated Fundamental matrix is the smallest. Geometric relationships are adjusted. Then, the three-dimensional shape (point cloud) is reconstructed based on the adjusted geometrical relationship. As a method of adjusting the error at this time, a processing method called Bundle Adjustment can be mentioned.

By the way, in the above-mentioned SfM, an error may occur in matching between feature points. In this case, feature point pairs are extracted between different feature points, and the accuracy of the restored three-dimensional shape is lowered. It ends up. Therefore, Patent Document 1 and Patent Document 2 disclose a system for correcting the positions of corresponding feature points.

Specifically, in the system disclosed in Patent Document 1, first, two images of the same object are arranged so that their epipolar lines are parallel to the scanning line, and each scanning line is arranged. The corresponding feature points are extracted from each image. Next, the system disclosed in Patent Document 1 accepts a user input of a line segment that specifies a corresponding portion on each image.

Then, the system disclosed in Patent Document 1 determines whether or not each corresponding feature point on the scanning line coincides with the intersection of the scanning line and the input line segment, and if they do not match. , Correct the position of each feature point to the position of the intersection. The system disclosed in Patent Document 1 then reconstructs the three-dimensional shape using the position-corrected feature points.

Further, the system disclosed in Patent Document 2 first extracts a combination of corresponding feature points from each pair of images obtained by photographing the same object. Next, the system disclosed in Patent Document 2 calculates a numerical matrix from the geometrical relationship between the line segments or points when the corresponding line segments or points are specified in each paired image.

Subsequently, in one image of the pair image, the feature point is p, the epipolar line in which the feature point p is present is l, and in the other image, the corresponding feature point is p'and the epipolar line in which the feature point p'is present. Let l'. In this case, the system disclosed in Patent Document 2 calculates the epipolar line l'corresponding to the epipolar line l using the calculated numerical matrix, and further, the distance d between the epipolar line l'and the feature point p'. Is calculated.

Then, the system disclosed in Patent Document 2 determines that the combination of the feature point p and the feature point p'is geometrically inconsistent when the distance d is equal to or greater than the threshold value. Then, the system disclosed in Patent Document 2 reconstructs the three-dimensional shape by using the calculated numerical matrix, excluding the combinations determined to have inconsistent geometric relationships.

As described above, according to the system disclosed in Patent Document 1 or Patent Document 2, the problem that the wrong feature point pair is extracted is solved, so that the deterioration of the accuracy of the restored three-dimensional shape is suppressed. It is thought that it will be done.

Japanese Patent No. 531465 International Publication No. 2019/065744

However, in the system disclosed in Patent Document 1, since the correction of the position of the feature point is performed only on the scan line along the epipolar line of the two images, the wrong feature point pair that is not on the same scan line is performed. Is not excluded.

On the other hand, in the system disclosed in Patent Document 2, a geometrical relationship between specified line intervals or points is obtained, and further, based on the distance between the epipolar line and the feature point obtained from this geometrical relationship. Since the wrong feature point pair is eliminated, the problem in Patent Document 1 is solved. However, the system disclosed in Patent Document 2 has a problem that the wrong feature point pair cannot be excluded when the distance between the epipolar line obtained from the geometrical relationship and the wrong feature point is short.

As described above, in the systems disclosed in Patent Document 1 and Patent Document 2, the elimination of wrong feature point pairs is insufficient, and the accuracy of the restored three-dimensional shape may decrease.

An example of an object of the present invention is an image processing apparatus, an image processing method, which can solve the above problem and suppress the extraction of an erroneous combination of feature points when extracting a combination of corresponding feature points from a plurality of images. And to provide a computer-readable recording medium.

In order to achieve the above object, the image processing apparatus according to one aspect of the present invention is an apparatus for constructing a three-dimensional shape of the object from a plurality of images of the object.
A feature point extraction unit that extracts a combination of corresponding feature points from each of the plurality of images,
When two or more line segments or two or more points corresponding to each other are specified on at least two images of the plurality of images, the geometrical relationship between the two or more line segments corresponding to each other, or each other. A matrix calculation unit that identifies a geometric relationship between two or more corresponding points and calculates a numerical matrix that expresses the specified geometric relationship.
Among the extracted combinations of the feature points, the unnecessary feature point identification unit that identifies the combination of the feature points in which the geometrical relationship between the feature points is inconsistent is used with the numerical matrix.
For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the feature points of the combination are calculated using the calculated camera matrix. A three-dimensional coordinate calculation unit that calculates the three-dimensional coordinates of the object corresponding to
For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates are compared with the two-dimensional coordinates of the one feature point. Based on the result, a conformity determination unit that determines whether or not the one feature point is appropriate, and
A three-dimensional shape construction unit that constructs a three-dimensional shape of the object using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
It is characterized by having.

Further, in order to achieve the above object, the image processing method in one aspect of the present invention is a method for constructing a three-dimensional shape of the object from a plurality of images of the object.
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two images of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. Or, a step that identifies a geometric relationship between two or more points corresponding to each other and calculates a numerical matrix that expresses the specified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the step and the step of identifying the combination of the feature points in which the geometrical relationship between the feature points is inconsistent.
(D) For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the combination is used by using the calculated camera matrix. The step of calculating the three-dimensional coordinates of the object corresponding to the feature point of
(E) For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point are calculated. Based on the comparison result with, the step of determining whether or not the one feature point is appropriate, and
(F) A step of constructing a three-dimensional shape of the object using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
It is characterized by having.

Further, in order to achieve the above object, the computer-readable recording medium in one aspect of the present invention is a computer that records a program for constructing a three-dimensional shape of the object from a plurality of images of the object by a computer. A readable recording medium
On the computer
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two images of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. Or, a step that identifies a geometric relationship between two or more points corresponding to each other and calculates a numerical matrix that expresses the specified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the step and the step of identifying the combination of the feature points in which the geometrical relationship between the feature points is inconsistent.
(D) For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the calculated camera matrix is used.
A step of calculating the three-dimensional coordinates of the object corresponding to the feature point of the combination, and
(E) For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point are calculated. Based on the comparison result with, the step of determining whether or not the one feature point is appropriate, and
(F) A step of constructing a three-dimensional shape of the object using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
It is characterized by recording a program including an instruction to execute.

As described above, according to the present invention, when extracting the corresponding combination of feature points from a plurality of images, it is possible to suppress the extraction of the wrong combination of feature points.

FIG. 1 is a block diagram schematically showing an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing a specific configuration of the image processing apparatus according to the embodiment of the present invention. FIG. 3 is a diagram showing an example of a plurality of images to be processed in the present embodiment. FIG. 4 is a diagram illustrating processing by the matrix calculation unit according to the embodiment of the present invention. FIG. 5 is a diagram illustrating processing by the shape building unit according to the embodiment of the present invention. FIG. 6 is a diagram showing an example of a pair image in which a combination of feature points is extracted. FIG. 7 is a diagram showing an example of the three-dimensional coordinates of the camera of the initial pair image and the rotation matrix obtained from the camera matrix. FIG. 8 is a diagram showing an example of a combination of a newly selected image after selection of an initial pair image and feature points extracted from the image. FIG. 9 is a diagram illustrating a process of reprojecting the three-dimensional coordinates of the feature points in the object onto the two-dimensional image. FIG. 10 is a flow chart showing the operation of the image processing apparatus according to the embodiment of the present invention. FIG. 11 is a block diagram showing an example of a computer that realizes the image processing apparatus according to the embodiment of the present invention.

(Embodiment)
Hereinafter, the image processing apparatus according to the embodiment of the present invention will be described with reference to FIGS. 1 to 11.

[Device configuration]
First, the schematic configuration of the image processing apparatus according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is a block diagram schematically showing an image processing apparatus according to an embodiment of the present invention.

The image processing device 10 in the present embodiment shown in FIG. 1 is a device for constructing a three-dimensional shape of an object from a plurality of images of the object. As shown in FIG. 1, the image processing apparatus 10 includes a feature point extraction unit 11, a matrix calculation unit 12, an unnecessary feature point identification unit 13, a three-dimensional coordinate calculation unit 14, a compatibility determination unit 15, and 3 It includes a three-dimensional shape construction unit 16.

The feature point extraction unit 11 extracts a combination of corresponding feature points from each of the plurality of images. The matrix calculation unit 12 determines the geometry of two or more line segments corresponding to each other when two or more line segments or two or more points corresponding to each other are specified on at least two images of the plurality of images. Identify the relationship, or the geometric relationship between two or more points that correspond to each other. Further, the matrix calculation unit 12 calculates a numerical matrix expressing the specified geometric relationship.

The unnecessary feature point identification unit 13 uses the calculated numerical matrix to identify the combinations of feature points whose geometric relationships between the feature points are inconsistent among the combinations of the extracted feature points. The three-dimensional coordinate calculation unit 14 calculates the camera matrix of each feature point constituting the combination for each feature point combination excluding the specified feature point combination. Further, the three-dimensional coordinate calculation unit 14 calculates the three-dimensional coordinates of the object corresponding to the feature points of the combination by using the calculated camera matrix for each combination of the feature points.

For each combination of feature points for which three-dimensional coordinates have been calculated, the suitability determination unit 15 applies a camera matrix corresponding to the image of the extraction source of one feature point constituting this combination to the three-dimensional coordinates of this combination. To do. Then, the suitability determination unit 15 calculates the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of this combination onto the image from which the above-mentioned one feature point is extracted by applying the camera matrix. Further, the suitability determination unit 15 compares the calculated two-dimensional coordinates with the two-dimensional coordinates of one feature point (two-dimensional coordinates on the image of the extraction source), and based on the comparison result, one of the above-mentioned ones. Determine if the feature points are appropriate.

The three-dimensional shape construction unit 16 constructs the three-dimensional shape of the object by using the three-dimensional coordinates of the object corresponding to the feature points of the combination determined to be appropriate by the suitability determination unit 15.

As described above, in the present embodiment, first, the combination of contradictory feature points is excluded by the geometric relationship obtained from the specified line segment or point, and then the three-dimensional coordinates of the obtained object are obtained. Is projected onto a two-dimensional image to determine whether or not the feature points are appropriate. That is, in the present embodiment, the wrong feature points are eliminated twice, before and after the calculation of the three-dimensional coordinates. Therefore, according to the present embodiment, when extracting the combination of the corresponding feature points from the plurality of images, it is possible to suppress the extraction of the wrong combination of the feature points. As a result, a highly accurate three-dimensional shape is constructed.

Subsequently, the configuration of the image processing device 10 according to the present embodiment will be described more specifically with reference to FIGS. 2 to 9. FIG. 2 is a block diagram showing a specific configuration of the image processing apparatus according to the embodiment of the present invention.

As shown in FIG. 2, in the present embodiment, the image processing apparatus 10 includes a feature point extraction unit 11, a matrix calculation unit 12, an unnecessary feature point identification unit 13, a three-dimensional coordinate calculation unit 14, and a compatibility determination unit 15. In addition to the three-dimensional shape construction unit 16, an image acquisition unit 17, a filtering unit 18, an input reception unit 19, and a display unit 20 are further provided. In FIG. 2, reference numeral 21 denotes a display device.

The image acquisition unit 17 acquires image data of each of a plurality of images showing a three-dimensional shape construction target from an external device, for example, an image pickup device, a terminal device, a storage device that holds image data, and the like. FIG. 3 is a diagram showing an example of a plurality of images to be processed in the present embodiment. In the example of FIG. 3, a pair image is illustrated, but the number of target images in the present embodiment is not particularly limited.

In the present embodiment, the feature point extraction unit 11 calculates, for example, a SIFT feature amount or a SURF feature amount for each image to specify the feature points, and further, the feature points corresponding to each other are supported between the images. Extract as a combination of feature points. In the example of FIG. 3, since there are two images, the combination of feature points is a feature point pair. In FIG. 3, the circled portion is one of the feature points. Further, the two images from which the combination of the corresponding feature points is extracted are hereinafter referred to as "pair images".

The filtering unit 18 calculates the geometrical relationship between the feature points for each combination of the feature points, identifies an erroneous feature point combination based on the calculation result, and eliminates the specified feature point combination. .. The filtering process by the filtering unit 18 is performed by using robust estimation as in the conventional SfM. In the present embodiment, in addition to the filtering by the unnecessary feature point specifying unit 13 described later, the filtering by the filtering unit 18 is also performed, so that the wrong combination of feature points is more reliably eliminated.

When the line segment or point corresponding to each other is designated on each image to be processed, the input receiving unit 19 accepts the input of the designated line segment or point. Further, when the input receiving unit 19 receives the input of the line segments corresponding to each other on each image, the input receiving unit 19 notifies the matrix calculation unit 12 of the information (coordinates of the start point and the end point) of each line segment. Further, when the input receiving unit 19 receives the input of the points corresponding to each other on each image, the input receiving unit 19 notifies the matrix calculation unit 12 of the information (coordinates) of each point. The line segment or the point may be designated by the user of the image processing device 10 or by another computer.

When the input receiving unit 19 notifies the information of the line segment or the point on each image, the matrix calculation unit 12 identifies and specifies the geometrical relationship between the line segment or the point based on the notified information. A numerical matrix that expresses the geometrical relationship is calculated, and this numerical matrix is defined as an absolute numerical matrix. The matrix calculation unit 12 cannot calculate a numerical matrix from the line segment or point information on each image notified from the input reception unit 19, or if the line segment or point information does not exist depending on the image, the feature point It is also possible to calculate a numerical matrix from a combination of feature points with a small error remaining by filtering after being extracted by the extraction unit 11. However, the numerical matrix calculated in this case is the same numerical matrix as the conventional one, and is not an absolute numerical matrix.

Here, the processing in the matrix calculation unit 12 will be described more specifically with reference to FIG. FIG. 4 is a diagram illustrating processing by the matrix calculation unit according to the embodiment of the present invention. In FIG. 4, E represents a epipolar plane, O _i denotes the camera center position of one image, O _'i denotes the center position of the camera of the other image. Further, in FIG. 4, the parallelograms shown on the left and right indicate the frame of the image, respectively.

Further, in the example of FIG. 4, a line segment L is designated in one image and a line segment L'is designated in the other image, and both line segments correspond to each other. In this case, in the line segment L, the epipolar line l _{n intersecting the start point and the epipolar line l n + m} intersecting at the end point are defined as absolute epipolar lines, and in the line segment L', the epipolar line l intersecting the start point is also defined. ' _{N and the epipolar line l'n + m} that intersects at the end point are defined as absolute epipolar lines.

Furthermore, the intersection x _i between the line segment L and the absolute epipolar line l _n, the intersection x _'i and _n' absolute epipolar line l and 'line segment L, defined as a combination of absolute characteristic points .. Further, the intersection x _j of the line segment L and the absolute epipolar line l _{n + m,} also the intersection x _'j' and absolute epipolar line l 'segment L and _{n + m,} is defined as a combination of absolute feature points.

As shown in FIG. 4, arbitrary epipolar lines l _{n + 1} and l _{n + 2} may be set _{between the epipolar line l n} and the epipolar line l _{n + m.} In this case, similarly, arbitrary epipolar lines l' _{n + 1} and l' _{n + 2} are set _{between the epipolar line l'n} and the epipolar line l' _{n + m.} Further, in this case, the intersection of the line segment L and the newly set epipolar lines l _{n + 1} and l _{n + 2} is the intersection of the line segment L'and the newly set epipolar lines l' _{n + 1} and l' _{n + 2,} respectively. It is a combination of absolute feature points. The interval between epipolar lines is set to an arbitrary value.

Therefore, in the example of FIG. 4, the matrix calculation unit 12 obtains an absolute combination of feature points as the geometric relationship between the lines, and uses the obtained combination of feature points to form the relationship of the following equation 1. From the equation, a Fundamental matrix (reference) is calculated as an absolute numerical matrix. In the following equation 1, "x" is a two-dimensional point obtained by projecting a point X in a three-dimensional space onto one image. "X'" is a two-dimensional point obtained by projecting a point X in a three-dimensional space onto the other image. T is the transposed matrix. Also, F is a Fundamental matrix. Further, in the present embodiment, the numerical matrix is not limited to the Fundamental matrix, and any matrix that can express a geometric relationship may be used.

References: Richard Hartley and Andrew Zisserman, “Multiple View Geometry in Computer Vision Second Edition”, Cambridge University Press, March 2004.

In the present embodiment, the unnecessary feature point identification unit 13 first uses a numerical matrix (Fundamental matrix) calculated by the matrix calculation unit 12 to combine feature points extracted by the feature point extraction unit 11 (filtering unit). From (excluding those excluded in 18), the combination of feature points with inconsistent geometric relationships is identified.

Further, the unnecessary feature point identification unit 13 is a feature point that overlaps with a designated line segment or point from the combination of feature points extracted by the feature point extraction unit 11 (excluding those excluded by the filtering unit 18). The combination of the feature points including the specified feature points can also be specified as the feature points whose geometric relationships are inconsistent. This is because if there is no feature point corresponding to one line segment or a feature point that overlaps only one point on the other line segment or the other point, this combination of feature points may be incorrect. Because it is expensive.

Here, the process in the unnecessary feature point specifying unit 13 will be described more specifically with reference to FIG. FIG. 5 is a diagram illustrating processing by the shape building unit according to the embodiment of the present invention. In FIG. 5, those with the reference numerals shown in FIG. 4 indicate those with the same symbols in FIG. Then, in FIG. 5, the combination of the feature point p _i and the feature point p _'i is assumed to be a determination of whether the target is wrong.

In the example of FIG. 5, unnecessary feature point specifying unit 13, using the relationship shown in Equation 2 below for the epipolar line, the epipolar line l _'1 corresponding to the epipolar lines l ₁ to the feature point p _i is present Is calculated.

Then, unnecessary feature point specifying unit 13, if the calculated epipolar line l _'1 and the feature point p' the distance d between the _i may determine whether a threshold value or more, the distance d is greater than or equal to a threshold value It determines that the combination of the feature point p _i and the feature point p _'i is wrong. In this case, unnecessary feature point specifying unit 13, a combination of the feature point p _i and the feature point p _'i, the geometric relationship is specified as a combination of feature points inconsistent.

The three-dimensional coordinate calculation unit 14 calculates the three-dimensional coordinates for constructing the three-dimensional shape by using the combination of the feature points not specified by the unnecessary feature point identification unit 13. Further, at this time, the three-dimensional coordinate calculation unit 14 can also use the corresponding point or the point on the corresponding line segment received by the input receiving unit 19 as a combination of the feature points.

Here, the calculation process of the three-dimensional coordinates by the three-dimensional coordinate calculation unit 14 will be specifically described with reference to FIGS. 6 to 8. FIG. 6 is a diagram showing an example of a pair image in which a combination of feature points is extracted. FIG. 7 is a diagram showing an example of the three-dimensional coordinates of the camera of the initial pair image and the rotation matrix obtained from the camera matrix. FIG. 8 is a diagram showing an example of a combination of a newly selected image after selection of an initial pair image and feature points extracted from the image.

As shown in FIG. 6, the three-dimensional coordinate calculation unit 14 first selects an image 31 and an image 32 as a pair of images (initial pair images). In this case, the feature points (m ₁ to m ₅ ) extracted from the image 31 correspond to the feature points (m ' ₁ to m ' _{5) extracted from the image 32.} m ₁ and m _{'1, m 2} and m' _{2, m 3} and m _{'3, m 4} and m' _{4, m 5} and m _'5 are each also referred to as a combination of feature points (hereinafter "feature point pair" To do). Further, in the example of FIG. 6, the image 31 is photographed by the camera 41, and the image 32 is photographed by the camera 42. In FIG. 6, M (M ₁ to M ₅ ) is a three-dimensional coordinate on the object corresponding to each feature point.

Subsequently, the three-dimensional coordinate calculation unit 14, using the initial pair images extracted feature point pair from each _{(m 1 ~ m 5, m} '1 ~ m' 5), the camera 41 captures an image 31 Camera The matrix P and the camera matrix P'of the camera 42 that captured the image 32 are calculated. Further, the camera matrix P and the camera matrix P'can be represented by the following equations 3 and 4, respectively, with the position of the camera 41 as the origin.

In the above equation 3, I is a rotation matrix of the camera 41. As shown in FIG. 7, since the position of the camera 41 is the origin, I = (1,1,1). _{Further, in the above equation} 4, R is a rotation matrix of the camera 42 (R = (R x, R _y , R _z )). As described above, t is a parallel traveling matrix and corresponds to the three-dimensional coordinates of the position of the camera 42 (t = (t _x , _ty , t _z )).

Therefore, in this case, R and t can be calculated by back-calculating from the camera matrix P and the camera matrix P'. Specifically, the three-dimensional coordinate calculation unit 14 calculates R and t by solving the equations shown in the following equations 5 to 7 using the coordinates of each feature point. In the equations 5 to 7, the m hat is the coordinates on the image A obtained by normalizing _{m (m 1} to m _5). Similarly, m 'hat, m' is the coordinates on the obtained image B the _{(m '1 ~ m' 5} ) is normalized. E is the Essential matrix and K is the camera calibration matrix.

Further, the calibration matrix K can be obtained from the following equations 8 and 9. Note that c _x and _cy are the center coordinates of the camera.

_{Next, the three-dimensional coordinate calculation unit 14 calculates the three-dimensional coordinates M (M 1} to M ₅ ) of the extracted feature points using the three-dimensional coordinates of the position of each camera and the rotation matrix. Specifically, the three-dimensional coordinate calculation unit 14 calculates the three-dimensional coordinate M by solving the following equation tens. Further, the matrix A in the equation 10 is represented by the equation 11. In Equation ^{11, p iT} is the line of camera matrix P, p ^'iT are camera matrix P' is a row of.

Next, as shown in FIG. 8, the three-dimensional coordinate calculation unit 14 newly selects one image 33 from the images obtained by extracting the feature points and other than the initial pair image. The newly selected image 33 and one of the initial pair images are used as a new pair image. The image 33 is taken by the camera 43.

Then, the three-dimensional coordinate calculation unit 14 identifies characteristic points of the image 33 corresponding to the feature points of the image 32 _{(m '' 1 ~ m '} ' 3), feature points between the feature point and the image 33 of the image 32 And is a feature point pair. Then, the three-dimensional coordinate calculation unit 14 calculates the camera matrix Pn of the camera 43 that captured the image 33. The camera matrix Pn can be represented by the following number 12.

Specifically, the three-dimensional coordinate calculation unit 14 calculates Rn and tun of the camera matrix Pn of the camera 43 by solving the equation shown in the following equation 13 using the specified feature points of the image 33. ..

In Equation 13, M _i is the 3-dimensional coordinates of the feature points in common with the image 32 in the newly selected image 33. The _mi hat is the normalized coordinates of the feature points in the newly selected image 33. The di indicates the distance between the camera 43 that captured the image 33 and the _{mi hat, as shown in the following number 14.}

Next, the three-dimensional coordinate calculation unit 14 uses the calculated Rn and tun of the camera matrix Pn of the camera 43 to obtain the three-dimensional coordinates _{of the specified feature points (m ″ 1} to m ″ _{3) of the image 33.} to calculate the M _i. In this case as well, the three-dimensional coordinate calculation unit 14 calculates the three-dimensional coordinates M (M ₁ to M ₃ ) of the feature points by solving the above equation 10. Through the above processing, the three-dimensional coordinate calculation unit 14 can calculate the three-dimensional coordinates of the object.

In the present embodiment, the conformity determination unit 15 uses, for example, three-dimensional coordinates obtained from one of the feature point pairs and a camera matrix corresponding to the image of the extraction source to obtain the original two-dimensional image. Reprojection is performed on the top, and the two-dimensional coordinates of the projected position are compared with the two-dimensional coordinates of the position at the time of extraction. Then, in the present embodiment, the suitability determination unit 15 calculates the difference between the former two-dimensional coordinates and the latter two-dimensional coordinates as a comparison result, and when the calculated difference is equal to or less than the threshold value, it becomes a target. It is judged that the feature points are appropriate. On the other hand, when the calculated difference exceeds the threshold value, the suitability determination unit 15 determines that the target feature point is not appropriate.

Further, in the present embodiment, the suitability determination unit 15 selects all or one part of the feature points constituting this combination for each combination of the feature points for which the three-dimensional coordinates have been calculated, and the selected feature points It is possible to determine whether or not they are appropriate one by one. For example, when the combination of feature points is composed of three feature points, the suitability determination unit 15 may determine whether or not each of the three feature points is appropriate, or only two of them. May be selected and it may be determined whether or not each of the two selected feature points is appropriate.

Here, the determination process by the conformity determination unit 15 will be specifically described with reference to FIG. FIG. 9 is a diagram illustrating a process of reprojecting the three-dimensional coordinates of the feature points in the object onto the two-dimensional image. Further, in the example of FIG. 9, an example of reprojecting one of the feature points on the image 32 shown in FIG. 6 is shown. Further, in FIG. 9, the three-dimensional coordinates corresponding to the feature points, that is, the three-dimensional coordinates in the world coordinate system are set to (X _W , Y _W , Z _W ), and the two-dimensional coordinates of the position at the time of extracting the feature points are set. _Let (x f, y _f ). Further, let the coordinates of the feature points in the camera coordinate system be (X _C , Y _C , Z _C ).

First, the conformity determination unit 15 uses the following equation 15 to obtain features from the camera matrix P'(= [R | t]) and the three-dimensional coordinates (X _W , Y _W , Z _W ) of the feature points. _{Calculate the coordinates (X C} , Y _C , Z _C ) of the points in the camera coordinate system.

_{Subsequently, the conformity determination unit 15 normalizes the coordinates (X C} , Y _C , Z _C ) of the feature points in the camera coordinate system using the following equation 16.

_{Subsequently, the conformity determination unit 15 uses the coordinates normalized by the above equation} 16 and the internal parameters of the camera (in the example of FIG. 9, the focal length f of the camera 42 and the coordinates of the image center position (c x, cy _). ) Is applied to the following equation 17 to calculate _{the two-dimensional coordinates (x p} , y _p ) when the feature points are reprojected on the image 32.

Next, the conformity judging unit 15 uses the number 18 below, the two-dimensional coordinates (x _{p, y p)} after re-projection calculated from the equation 17 and the two-dimensional coordinates of the position at the time of the feature point extraction The difference d from (x _f , y _{f) is calculated.}

After that, the suitability determination unit 15 determines that the target feature point is appropriate when the calculated difference d is equal to or less than the threshold value G, and when the calculated difference d exceeds the threshold value G, it becomes a target. It is judged that the feature points are not appropriate.

In the present embodiment, the conformity determination unit 15 excludes the three-dimensional coordinates corresponding to the feature points determined to be inappropriate and the combination of all the feature points corresponding to the three-dimensional coordinates. Further, in the present embodiment, the conformity determination unit 15 excludes only the feature points determined to be inappropriate, and sets the three-dimensional coordinates corresponding to the feature points determined to be inappropriate and the three-dimensional coordinates. Of the corresponding combinations of feature points, it is possible not to exclude feature points other than the feature points determined to be inappropriate. In this case, when the suitability determination of all the feature points is completed, the suitability determination unit 15 has less than two feature points corresponding to the three-dimensional coordinates for all the three-dimensional coordinates. Three-dimensional coordinates need to be excluded.

The three-dimensional shape construction unit 16 recalculates the three-dimensional coordinates using the remaining feature points, constructs the point cloud data of the object, and sets this as the three-dimensional shape of the object.

Further, the display unit 20 displays the three-dimensional shape constructed by the three-dimensional shape construction unit 16 on the screen of the display device 21. Specifically, the display unit 20 creates image data for displaying the constructed three-dimensional shape on the two-dimensional screen, and outputs the created image data to the display device 21.

Further, in the present embodiment, the combination of the feature points extracted by the feature point extraction unit 11 is used, for example, the combination of the feature points after only the filtering by the filtering unit 18 is performed, and the three-dimensional coordinates are used. The calculation unit 14 can also calculate the provisional three-dimensional coordinates of the object. In this case, the three-dimensional shape building unit 16 further builds a temporary three-dimensional shape using the calculated temporary three-dimensional coordinates. After that, the display unit 20 displays the constructed temporary three-dimensional shape on the screen of the display device 21. In this case, the user can specify line segments or points corresponding to each other in the temporary three-dimensional shape displayed on the screen.

[Device operation]
Next, the operation of the image processing device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 10 is a flow chart showing the operation of the image processing apparatus according to the embodiment of the present invention. In the following description, FIGS. 1 to 9 will be referred to as appropriate. Further, in the present embodiment, the image processing method is implemented by operating the image processing device 10. Therefore, the description of the image processing method in the present embodiment will be replaced with the following description of the operation of the image processing device 10.

As shown in FIG. 10, first, the image acquisition unit 17 acquires image data of each of a plurality of images showing a three-dimensional shape construction target from an external device (step A1).

Next, the feature point extraction unit 11 identifies the feature points in the image for each image data acquired in step A1, and further extracts the feature points corresponding to each other as a combination of the corresponding feature points. (Step A2).

Next, the filtering unit 18 calculates the geometrical relationship between the feature points for each combination of the feature points extracted in step A2, and identifies an erroneous feature point combination based on the calculation result. The combination of the specified feature points is excluded (step A3).

Next, when the filtering process according to step A3 is executed, the three-dimensional coordinate calculation unit 14 calculates temporary three-dimensional coordinates using the combination of the feature points after filtering, and further, the three-dimensional shape construction unit 16 Constructs a tentative 3D shape using the calculated tentative 3D coordinates. Further, the display unit 20 displays the constructed temporary three-dimensional shape on the screen of the display device 21 (step A4).

When step A4 is executed and the temporary three-dimensional shape is displayed on the screen of the display device 21, the user can use the line segments corresponding to each other on each image via the input device (not shown in FIG. 2). Or specify a point. As a result, the input receiving unit 16 receives the input of the designated line segment or point (step A5). Further, the input receiving unit 16 notifies the matrix calculation unit 12 of the information of each designated line segment or the information of each point.

Next, when step A5 is executed, the matrix calculation unit 12 identifies the geometric relationship between the line segments or the points based on the information of each line segment or the information of each point, and the specified geometric relationship. Is calculated (step A6).

Next, the unnecessary feature point identification unit 13 uses the numerical matrix (Fundamental matrix) calculated in step A6, and from the combination of feature points after the execution of step A3, the combination of feature points whose geometric relationships are inconsistent. (Step A7).

Next, the three-dimensional coordinate calculation unit 14 calculates the three-dimensional coordinates for constructing the three-dimensional shape by using the combination of the feature points not specified in step A7 (step A8).

Next, the suitability determination unit 15 sets a camera matrix corresponding to the image of the extraction source of one feature point constituting this combination for each combination of the feature points whose three-dimensional coordinates are calculated in step A8. Applies to the 3D coordinates of. Then, the suitability determination unit 15 calculates the two-dimensional coordinates obtained by projecting the three-dimensional coordinates of this combination onto the image from which the above-mentioned one feature point is extracted by applying the camera matrix. Further, the suitability determination unit 15 compares the calculated two-dimensional coordinates with the two-dimensional coordinates of one feature point (two-dimensional coordinates on the image of the extraction source), and based on the comparison result, one of the above-mentioned ones. It is determined whether or not the feature points are appropriate (step A9).

Next, the three-dimensional shape construction unit 16 constructs the three-dimensional shape of the object by using the three-dimensional coordinates of the object corresponding to the feature points of the combination determined to be appropriate in step A6 (. Step A10). After that, the display unit 20 displays the three-dimensional shape constructed in step A10 on the screen of the display device 21 (step A11).

[Effect of the embodiment]
As described above, in the image processing apparatus 10 of the present embodiment, erroneous feature points can be eliminated by the filtering unit 18, the unnecessary feature point identification unit 13, and the compatibility determination unit 15. Therefore, the image processing device 10 can suppress a situation in which an erroneous combination of feature points is extracted when extracting a combination of corresponding feature points from a plurality of images. As a result, according to the present embodiment, the three-dimensional shape of the object can be constructed with high accuracy.

[program]
The program in this embodiment may be any program that causes a computer to execute steps A1 to A11 shown in FIG. By installing this program on a computer and executing it, the image processing device 10 and the image processing method according to the present embodiment can be realized. In this case, the computer processor includes a feature point extraction unit 11, a matrix calculation unit 12, an unnecessary feature point identification unit 13, a three-dimensional coordinate calculation unit 14, a conformity determination unit 15, a three-dimensional shape construction unit 16, and an image acquisition unit 17. , It functions as a filtering unit 18 and an input receiving unit 19, and performs processing.

Further, the program in the present embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer has a feature point extraction unit 11, a matrix calculation unit 12, an unnecessary feature point identification unit 13, a three-dimensional coordinate calculation unit 14, a suitability determination unit 15, and a three-dimensional shape construction unit 16, respectively. It may function as any of the image acquisition unit 17, the filtering unit 18, and the input reception unit 19.

Here, a computer that realizes an image processing device by executing the program according to the present embodiment will be described with reference to FIG. FIG. 11 is a block diagram showing an example of a computer that realizes the image processing apparatus according to the embodiment of the present invention.

As shown in FIG. 11, the computer 110 includes a CPU 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. Each of these parts is connected to each other via a bus 121 so as to be capable of data communication. Further, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to the CPU 111 or in place of the CPU 111.

The CPU 111 executes various operations by expanding the program (code group) in the present embodiment stored in the storage device 113 into the main memory 112 and executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Further, the program according to the present embodiment is provided in a state of being stored in a computer-readable recording medium 120. The program in the present embodiment may be distributed on the Internet connected via the communication interface 117.

Further, specific examples of the storage device 113 include a semiconductor storage device such as a flash memory in addition to a hard disk drive. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader / writer 116 mediates the data transmission between the CPU 111 and the recording medium 120, reads the program from the recording medium 120, and writes the processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

Specific examples of the recording medium 120 include a general-purpose semiconductor storage device such as CF (CompactFlash (registered trademark)) and SD (SecureDigital), a magnetic recording medium such as a flexible disk, or a CD-. Examples include optical recording media such as ROM (CompactDiskReadOnlyMemory).

The image processing device 10 in the present embodiment can also be realized by using hardware (for example, a circuit) corresponding to each part instead of the computer in which the program is installed. Further, the image processing apparatus 10 may be partially realized by a program and the rest may be realized by hardware.

A part or all of the above-described embodiments can be expressed by the following descriptions (Appendix 1) to (Appendix 18), but the description is not limited to the following.

(Appendix 1)
A device for constructing a three-dimensional shape of an object from a plurality of images of the object.
A feature point extraction unit that extracts a combination of corresponding feature points from each of the plurality of images,
When two or more line segments or two or more points corresponding to each other are specified on at least two images of the plurality of images, the geometrical relationship between the two or more line segments corresponding to each other, or each other. A matrix calculation unit that identifies a geometric relationship between two or more corresponding points and calculates a numerical matrix that expresses the specified geometric relationship.
Among the extracted combinations of the feature points, the unnecessary feature point identification unit that identifies the combination of the feature points in which the geometrical relationship between the feature points is inconsistent is used with the numerical matrix.
For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the feature points of the combination are calculated using the calculated camera matrix. A three-dimensional coordinate calculation unit that calculates the three-dimensional coordinates of the object corresponding to
For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates are compared with the two-dimensional coordinates of the one feature point. Based on the result, a conformity determination unit that determines whether or not the one feature point is appropriate, and
A three-dimensional shape construction unit that constructs a three-dimensional shape of the object using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
An image processing device characterized by being equipped with.

(Appendix 2)
The image processing apparatus according to Appendix 1.
As a comparison result, the suitability determination unit calculates the difference between the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point, and when the calculated difference is equal to or less than the threshold value, the one feature point is determined. It is determined that the feature point is appropriate, and when the calculated difference exceeds the threshold value, it is determined that the one feature point is not appropriate.
An image processing device characterized by this.

(Appendix 3)
The image processing apparatus according to Appendix 1 or 2.
The suitability determination unit selects all or one part of the feature points constituting the combination for each combination of the feature points for which the three-dimensional coordinates have been calculated, and whether or not each of the selected feature points is appropriate. To judge,
An image processing device characterized by this.

(Appendix 4)
The image processing apparatus according to any one of Supplementary note 1 to 3.
The unnecessary feature point identification unit further sets a feature point that overlaps with two or more designated line segments or two or more points corresponding to each other among the feature points constituting the extracted combination of the feature points. Identify,
An image processing device characterized by this.

(Appendix 5)
The image processing apparatus according to any one of Appendix 1 to 4.
It further includes a display unit that displays the three-dimensional shape constructed by the three-dimensional shape construction unit on the screen.
An image processing device characterized by this.

(Appendix 6)
The image processing apparatus according to Appendix 5.
The three-dimensional coordinate calculation unit calculates temporary three-dimensional coordinates of the object by using the combination of the feature points extracted by the feature point extraction unit.
The three-dimensional shape construction unit constructs a temporary three-dimensional shape of the object by using the calculated temporary three-dimensional coordinates.
The display unit further displays the temporary three-dimensional shape on the screen.
An image processing device characterized by this.

(Appendix 7)
A method for constructing a three-dimensional shape of an object from a plurality of images of the object.
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two images of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. Or, a step that identifies a geometric relationship between two or more points corresponding to each other and calculates a numerical matrix that expresses the specified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the step and the step of identifying the combination of the feature points in which the geometrical relationship between the feature points is inconsistent.
(D) For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the combination is used by using the calculated camera matrix. The step of calculating the three-dimensional coordinates of the object corresponding to the feature point of
(E) For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point are calculated. Based on the comparison result with, the step of determining whether or not the one feature point is appropriate, and
(F) A step of constructing a three-dimensional shape of the object using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
An image processing method comprising.

(Appendix 8)
The image processing method described in Appendix 7
In the step (e), as a comparison result, the difference between the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point is calculated, and when the calculated difference is equal to or less than the threshold value, the one feature point. Is appropriate, and when the calculated difference exceeds the threshold value, it is determined that the one feature point is not appropriate.
An image processing method characterized by that.

(Appendix 9)
The image processing method according to Appendix 7 or 8.
In the step (e), for each combination of the feature points for which the three-dimensional coordinates have been calculated, all or one part of the feature points constituting the combination is selected, and whether the selected feature points are appropriate one by one. Judge whether
An image processing method characterized by that.

(Appendix 10)
The image processing method according to any one of Appendix 7 to 9.
In the step (c), among the feature points constituting the combination of the extracted feature points, the feature points overlapping the designated two or more line segments or two or more points corresponding to each other are designated. Identify,
An image processing method characterized by that.

(Appendix 11)
The image processing method according to any one of Appendix 7 to 10.
(G) Further having a step of displaying the three-dimensional shape constructed by the step of (f) on the screen.
An image processing method characterized by that.

(Appendix 12)
The image processing method according to Appendix 11,
(H) Using the combination of the feature points extracted in the step (a), the tentative three-dimensional coordinates of the object are calculated, and the calculated tentative three-dimensional coordinates are used to calculate the target. A step of constructing a temporary three-dimensional shape of an object and further displaying the temporary three-dimensional shape on the screen.
Have more
An image processing method characterized by that.

(Appendix 13)
A computer-readable recording medium in which a computer records a program for constructing a three-dimensional shape of an object from a plurality of images of the object.
On the computer
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two images of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. Or, a step that identifies a geometric relationship between two or more points corresponding to each other and calculates a numerical matrix that expresses the specified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the step and the step of identifying the combination of the feature points in which the geometrical relationship between the feature points is inconsistent.
(D) For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the combination is used by using the calculated camera matrix. The step of calculating the three-dimensional coordinates of the object corresponding to the feature point of
(E) For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point are calculated. Based on the comparison result with, the step of determining whether or not the one feature point is appropriate, and
(F) A step of constructing a three-dimensional shape of the object using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
A computer-readable recording medium characterized by recording a program, including instructions for executing the program.

(Appendix 14)
The computer-readable recording medium according to Appendix 13, which is a computer-readable recording medium.
In the step (e), as a comparison result, the difference between the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point is calculated, and when the calculated difference is equal to or less than the threshold value, the one feature point. Is appropriate, and when the calculated difference exceeds the threshold value, it is determined that the one feature point is not appropriate.
A computer-readable recording medium characterized by that.

(Appendix 15)
A computer-readable recording medium according to Appendix 13 or 14.
In the step (e), for each combination of the feature points for which the three-dimensional coordinates have been calculated, all or one part of the feature points constituting the combination is selected, and whether the selected feature points are appropriate one by one. Judge whether
A computer-readable recording medium characterized by that.

(Appendix 16)
A computer-readable recording medium according to any one of Appendix 13 to 15.
In the step (c), among the feature points constituting the combination of the extracted feature points, the feature points overlapping the designated two or more line segments or two or more points corresponding to each other are designated. Identify,
A computer-readable recording medium characterized by that.

(Appendix 17)
A computer-readable recording medium according to any one of Appendix 13 to 16.
In addition to the computer
(G) The three-dimensional shape constructed by the step (f) is displayed on the screen, and the step is executed.
A computer-readable recording medium characterized by that.

(Appendix 18)
The computer-readable recording medium according to Appendix 17, wherein the recording medium is readable.
In addition to the computer
(H) Using the combination of the feature points extracted in the step (a), the tentative three-dimensional coordinates of the object are calculated, and the calculated tentative three-dimensional coordinates are used to calculate the target. A temporary three-dimensional shape of an object is constructed, and the temporary three-dimensional shape is displayed on the screen, and a step is executed.
A computer-readable recording medium characterized by that.

Although the invention of the present application has been described above with reference to the embodiment, the invention of the present application is not limited to the above embodiment. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the structure and details of the present invention.

This application claims priority based on Japanese application Japanese Patent Application No. 2019-189792 filed on October 16, 2019, and incorporates all of its disclosures herein.

As described above, according to the present invention, when extracting the corresponding combination of feature points from a plurality of images, it is possible to suppress the extraction of the wrong combination of feature points. The present invention is useful in a technique for constructing a three-dimensional shape from a plurality of images such as SfM.

10 Image processing unit 11 Feature point extraction unit 12 Matrix calculation unit 13 Unnecessary feature point identification unit 14 3D coordinate calculation unit 15 Conformity judgment unit 16 3D shape construction unit 17 Image acquisition unit 18 Filtering unit 19 Input reception unit 20 Display unit 21 Display device 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (18)

1. A device for constructing a three-dimensional shape of an object from a plurality of images of the object.
   A feature point extraction means for extracting a combination of corresponding feature points from each of the plurality of images,
   When two or more line segments or two or more points corresponding to each other are specified on at least two images of the plurality of images, the geometrical relationship between the two or more line segments corresponding to each other, or each other. A matrix calculation means that identifies a geometric relationship between two or more corresponding points and calculates a numerical matrix that expresses the specified geometric relationship.
   An unnecessary feature point identifying means for identifying the combination of the feature points whose geometric relationship between the feature points is inconsistent among the extracted combinations of the feature points using the numerical matrix.
   For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the feature points of the combination are calculated using the calculated camera matrix. A three-dimensional coordinate calculation means for calculating the three-dimensional coordinates of the object corresponding to
   For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates are compared with the two-dimensional coordinates of the one feature point. A conformity determination means for determining whether or not the one feature point is appropriate based on the result,
   A three-dimensional shape constructing means for constructing a three-dimensional shape of the object using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
   An image processing device characterized by being equipped with.
2. The image processing apparatus according to claim 1.
   As a comparison result, the suitability determination means calculates the difference between the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point, and when the calculated difference is equal to or less than the threshold value, the one feature point is determined. It is determined that the feature point is appropriate, and when the calculated difference exceeds the threshold value, it is determined that the one feature point is not appropriate.
   An image processing device characterized by this.
3. The image processing apparatus according to claim 1 or 2.
   Whether or not the suitability determination means selects all or one part of the feature points constituting the combination for each combination of the feature points for which the three-dimensional coordinates have been calculated, and is appropriate for each of the selected feature points. To judge,
   An image processing device characterized by this.
4. The image processing apparatus according to any one of claims 1 to 3.
   The unnecessary feature point identifying means further obtains feature points that overlap with two or more designated line segments or two or more points corresponding to each other among the feature points constituting the extracted combination of the feature points. Identify,
   An image processing device characterized by this.
5. The image processing apparatus according to any one of claims 1 to 4.
   Further provided with a display means for displaying the three-dimensional shape constructed by the three-dimensional shape construction means on the screen.
   An image processing device characterized by this.
6. The image processing apparatus according to claim 5.
   The three-dimensional coordinate calculation means calculates provisional three-dimensional coordinates of the object by using the combination of the feature points extracted by the feature point extraction means.
   The three-dimensional shape construction means constructs a temporary three-dimensional shape of the object by using the calculated temporary three-dimensional coordinates.
   The display means further displays the temporary three-dimensional shape on the screen.
   An image processing device characterized by this.
7. A method for constructing a three-dimensional shape of an object from a plurality of images of the object.
   A combination of corresponding feature points is extracted from each of the plurality of images.
   When two or more line segments or two or more points corresponding to each other are specified on at least two images of the plurality of images, the geometrical relationship between the two or more line segments corresponding to each other, or each other. The geometric relationship between two or more corresponding points is specified, and a numerical matrix expressing the specified geometric relationship is calculated.
   Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are identified.
   For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the feature points of the combination are calculated using the calculated camera matrix. Calculate the three-dimensional coordinates of the object corresponding to
   For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates are compared with the two-dimensional coordinates of the one feature point. Based on the result, it is determined whether or not the one feature point is appropriate, and
   An image processing method characterized in that a three-dimensional shape of the object is constructed by using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate.
8. The image processing method according to claim 7.
   In the determination of whether or not the one feature point is optimized, as a comparison result, the difference between the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point is calculated, and when the calculated difference is equal to or less than the threshold value, It is determined that the one feature point is appropriate, and when the calculated difference exceeds the threshold value, it is determined that the one feature point is not appropriate.
   An image processing method characterized by that.
9. The image processing method according to claim 7 or 8.
   In the determination of whether or not the one feature point is optimized, all or one part of the feature points constituting the combination is selected for each combination of the feature points for which the three-dimensional coordinates are calculated, and the selected feature points are obtained. Judge whether it is appropriate one by one,
   An image processing method characterized by that.
10. The image processing method according to any one of claims 7 to 9.
    In specifying the combination of the feature points, further, among the feature points constituting the extracted combination of the feature points, the feature points overlapping the designated two or more line segments or two or more points corresponding to each other. To identify,
    An image processing method characterized by that.
11. The image processing method according to any one of claims 7 to 10.
    The three-dimensional shape constructed by constructing the three-dimensional shape is displayed on the screen.
    An image processing method characterized by that.
12. The image processing method according to claim 11.
    Using the combination of the feature points extracted by extracting the combination of the corresponding feature points, the tentative three-dimensional coordinates of the object are calculated, and the calculated tentative three-dimensional coordinates are used to calculate the target. A temporary three-dimensional shape of an object is constructed, and the temporary three-dimensional shape is displayed on the screen.
    An image processing method characterized by that.
13. A computer-readable recording medium for constructing a three-dimensional shape of an object from a plurality of images of the object by a computer.
    On the computer
    A combination of corresponding feature points is extracted from each of the plurality of images.
    When two or more line segments or two or more points corresponding to each other are specified on at least two images of the plurality of images, the geometrical relationship between the two or more line segments corresponding to each other, or each other. The geometric relationship between two or more corresponding points is specified, and a numerical matrix expressing the specified geometric relationship is calculated.
    Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are identified.
    For each combination of the feature points excluding the specified combination of the feature points, a camera matrix of each feature point constituting the combination is calculated, and further, the feature points of the combination are calculated using the calculated camera matrix. To calculate the three-dimensional coordinates of the object corresponding to
    For each combination of the feature points for which the three-dimensional coordinates have been calculated, the camera matrix corresponding to the image of the extraction source of one feature point constituting the combination is applied to the three-dimensional coordinates of the combination. The two-dimensional coordinates obtained by projecting the three-dimensional coordinates of the combination onto the image from which the one feature point is extracted are calculated, and the calculated two-dimensional coordinates are compared with the two-dimensional coordinates of the one feature point. Based on the result, it is determined whether or not the one feature point is appropriate, and the result is determined.
    Using the three-dimensional coordinates of the object corresponding to the feature points determined to be appropriate, the three-dimensional shape of the object is constructed.
    Recording the program containing the instructions,
    A computer-readable recording medium characterized by that.
14. The computer-readable recording medium according to claim 13.
    In the determination of whether or not the one feature point is optimized, as a comparison result, the difference between the calculated two-dimensional coordinates and the two-dimensional coordinates of the one feature point is calculated, and when the calculated difference is equal to or less than the threshold value, It is determined that the one feature point is appropriate, and when the calculated difference exceeds the threshold value, it is determined that the one feature point is not appropriate.
    A computer-readable recording medium characterized by that.
15. A computer-readable recording medium according to claim 13 or 14.
    In the determination of whether or not the one feature point is optimized, all or one part of the feature points constituting the combination is selected for each combination of the feature points for which the three-dimensional coordinates are calculated, and the selected feature points are obtained. Judge whether it is appropriate one by one,
    A computer-readable recording medium characterized by that.
16. A computer-readable recording medium according to any one of claims 13 to 15.
    In specifying the combination of the feature points, further, among the feature points constituting the extracted combination of the feature points, the feature points overlapping the designated two or more line segments or two or more points corresponding to each other. To identify,
    A computer-readable recording medium characterized by that.
17. A computer-readable recording medium according to any one of claims 13 to 16.
    The program is on the computer
    The three-dimensional shape constructed by constructing the three-dimensional shape is displayed on the screen.
    A computer-readable recording medium characterized by that.
18. The computer-readable recording medium according to claim 17.
    The program is on the computer
    Using the combination of the feature points extracted by extracting the combination of the corresponding feature points, the tentative three-dimensional coordinates of the object are calculated, and the calculated tentative three-dimensional coordinates are used to calculate the target. A temporary three-dimensional shape of an object is constructed, and the temporary three-dimensional shape is displayed on the screen.
    A computer-readable recording medium characterized by that.

Images