JP6989153B2 - Image processing equipment, image processing methods, and programs - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method for enabling the construction of a three-dimensional shape from a plurality of images, and further relates to a program for realizing these.

In recent years, attention has been paid to a technique for constructing a three-dimensional shape of an object shown in an image. SfM (Structure from Motion) is known as a typical example of such a technique. In SfM, a specific object is photographed a plurality of times while changing the viewpoint of the camera, and the three-dimensional shape of the specific object is restored from the obtained plurality of images.

Specifically, first, in SfM, the feature amount (for example, SIFT feature amount, SURF feature amount) is calculated for each image, and robustness feature points that are resistant to image enlargement / reduction, rotation, and illuminance change are extracted. Will be done. Next, matching of the extracted feature points is executed between the images, and a pair of matching feature points is extracted. Robust estimation, for example, then calculates the geometric relationships of the feature point pairs and excludes the erroneous feature point pairs.

Then, for each of several feature point pairs, a Fundamental matrix is calculated based on these geometric relationships, and for each feature point pair so that the difference between each calculated Fundamental matrix is the smallest. Geometric relationships are adjusted. Then, the three-dimensional shape (point cloud) is reconstructed based on the adjusted geometrical relationship. As a method of adjusting the error at this time, a processing method called Bundle Adjustment can be mentioned.

By the way, in the above-mentioned SfM, an error may occur in matching, and in this case, a pair is extracted between different feature points, and the accuracy of the restored three-dimensional shape is lowered. In addition, there may be a region where the number of extracted feature point pairs is small or the matching is not successful, resulting in a small number of feature point pairs. In this case as well, the accuracy of the restored three-dimensional shape is reduced.

On the other hand, Patent Document 1 discloses a system for correcting the positions of corresponding feature points. Specifically, in the system disclosed in Patent Document 1, first, two images are arranged so that their epipolar lines are parallel to the scanning line, and the corresponding feature points from each image are set for each scanning line. Extract.

The system then accepts a user input of a line segment that specifies a corresponding location on each image. The system then determines whether each corresponding feature point on the scan line coincides with the intersection of this scan line with the input line segment, and if not, determines the position of each feature point. Correct to the position of the intersection. The system then performs a 3D shape reconstruction using the position-corrected feature points.

As described above, according to the system disclosed in Patent Document 1, the problem that an erroneous feature point pair is extracted and the problem that a region with a small number of feature point pairs is generated are solved, so that the system is restored. The problem that the accuracy of the three-dimensional shape is reduced is solved.

Japanese Patent No. 531465

However, in the system disclosed in Patent Document 1, the correction of the position of the feature point is performed only on the scan line along the epipolar line of the two images, so that the wrong feature point pair that is not on the same scan line is performed. Will not be corrected for. Therefore, in the system disclosed in Patent Document 1, the correction of the wrong feature point pair is insufficient, and it becomes difficult to suppress the deterioration of the accuracy of the restored three-dimensional shape.

An example of an object of the present invention is an image processing apparatus, an image processing method, which can solve the above problem and suppress extraction of an erroneous combination of feature points when extracting a combination of corresponding feature points from a plurality of images. And to provide the program.

In order to achieve the above object, the image processing apparatus according to one aspect of the present invention is an apparatus for constructing a three-dimensional shape from a plurality of images.
A feature point extraction unit that extracts a combination of corresponding feature points from each of the plurality of images,
When two or more line segments or two or more points corresponding to each other are specified on at least two of the plurality of images, the geometrical relationship between the two or more line segments corresponding to each other, or each other. A matrix calculation unit that identifies a geometric relationship between two or more corresponding points and calculates a numerical matrix that expresses the specified geometric relationship.
Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are specified, and the feature points other than the specified combinations of the feature points are specified. The shape building unit that builds the three-dimensional shape using the combination of
It is characterized by having.

Further, in order to achieve the above object, the image processing method in one aspect of the present invention is a method for constructing a three-dimensional shape from a plurality of images.
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. , Or a step that identifies a geometric relationship between two or more points that correspond to each other and calculates a numerical matrix that expresses the identified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are specified, and the combinations of the feature points other than the specified combinations are specified. A step of constructing the three-dimensional shape using the combination of the feature points, and
It is characterized by having.

Furthermore, in order to achieve the above object, a program according to an aspect of the present invention, by a computer, a program for constructing a three-dimensional shape from a plurality of images,
To the computer
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. , Or a step that identifies a geometric relationship between two or more points that correspond to each other and calculates a numerical matrix that expresses the identified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are specified, and the combinations of the feature points other than the specified combinations are specified. A step of constructing the three-dimensional shape using the combination of the feature points, and
Allowed to run and wherein the Turkey.

As described above, according to the present invention, when extracting the combination of corresponding feature points from a plurality of images, it is possible to suppress the extraction of the wrong combination of feature points.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing a specific configuration of the image processing apparatus according to the embodiment of the present invention. FIG. 3 is a diagram showing an example of a plurality of images to be processed in the present embodiment. FIG. 4 is a diagram illustrating processing by the matrix calculation unit according to the embodiment of the present invention. FIG. 5 is a diagram illustrating processing by the shape building unit according to the embodiment of the present invention. FIG. 6 is a flow chart showing the operation of the image processing device according to the embodiment of the present invention. FIG. 7 is a block diagram showing an example of a computer that realizes the image processing apparatus according to the embodiment of the present invention.

(Embodiment)
Hereinafter, the image processing apparatus, the image processing method, and the program according to the embodiment of the present invention will be described with reference to FIGS. 1 to 7.

[Device configuration]
First, the schematic configuration of the image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention.

As shown in FIG. 1, the image processing device 10 in the present embodiment is a device for constructing a three-dimensional shape from a plurality of images. Image data of a plurality of images to be processed is input to the image processing device 10.

Further, as shown in FIG. 1, the image processing apparatus 10 includes a feature point extraction unit 11, a matrix calculation unit 12, and a shape construction unit 13. Of these, the feature point extraction unit 11 extracts a combination of corresponding feature points from each of the plurality of images.

When the matrix calculation unit 12 specifies two or more line segments or two or more points corresponding to each other on at least two images out of a plurality of images, the geometry of the two or more line segments corresponding to each other is specified. Identify the relationship, or the geometric relationship between two or more points that correspond to each other. Then, the matrix calculation unit 12 calculates a numerical matrix expressing the specified geometric relationship.

The shape construction unit 13 uses the numerical matrix calculated by the matrix calculation unit 12 to specify a combination of feature points whose geometric relationships between the feature points are inconsistent among the combinations of the extracted feature points. Then, the shape building unit 13 constructs a three-dimensional shape by using a combination of feature points other than the specified combination of feature points.

Thus, in the present embodiment, since the wrong combination of feature points is specified from the geometric relationship specified from the specified two or more line segments or points, the wrong combination that is not on the same scanning line is wrong. The combination of feature points can also be excluded. According to the present embodiment, when extracting a combination of corresponding feature points from a plurality of images, it is possible to suppress the extraction of a combination of wrong feature points and improve the accuracy of the restored three-dimensional shape.

Subsequently, the configuration of the image processing apparatus 10 in the present embodiment will be described more specifically with reference to FIGS. 2 to 5. FIG. 2 is a block diagram showing a specific configuration of the image processing apparatus according to the embodiment of the present invention.

As shown in FIG. 2, in the present embodiment, the image processing apparatus 10 includes an image acquisition unit 14 and a filtering unit 15 in addition to the feature point extraction unit 11, the matrix calculation unit 12, and the shape construction unit 13. It further includes an input receiving unit 16 and a display unit 17.

The image acquisition unit 14 acquires image data of each of a plurality of images showing a three-dimensional shape construction target from an external device, for example, an image pickup device, a terminal device, a storage device holding image data, and the like. FIG. 3 is a diagram showing an example of a plurality of images to be processed in the present embodiment. In the example of FIG. 3, a pair image is exemplified, but the number of target images in the present embodiment is not particularly limited.

In the present embodiment, the feature point extraction unit 11 calculates, for example, a SIFT feature amount or a SURF feature amount for each image to specify the feature points, and further, the feature points corresponding to each other are supported between the images. Extract as a combination of feature points. In the example of FIG. 3, since there are two images, the combination of feature points is a feature point pair. Further, in FIG. 3, the portion circled is one of the feature points.

The filtering unit 15 calculates the geometrical relationship between the feature points for each combination of the feature points, identifies an erroneous feature point combination based on the calculation result, and eliminates the specified feature point combination. .. The filtering process by the filtering unit 15 is performed by using robust estimation as in the conventional case. In the present embodiment, in addition to the filtering by the shape building unit 13 described later, the filtering by the filtering unit 15 is also performed, so that the wrong combination of feature points is more reliably eliminated.

When the line segment or point corresponding to each other is designated on each image to be processed, the input receiving unit 16 accepts the input of the designated line segment or point. Further, the input receiving unit 16 receives the information of each line segment (coordinates of the start point and the end point) when the input of the line segments corresponding to each other is received on each image, and each point when the input of the points corresponding to each other is received. Information (coordinates) of the above is notified to the matrix calculation unit 12. The line segment or the point may be designated by the user of the image processing apparatus 10 or by another computer.

When the matrix calculation unit 12 is notified of the information of the line segment or point on each image from the input reception unit 16, the matrix calculation unit 12 identifies and specifies the geometrical relationship between the line segment or the point based on the notified information. A numerical matrix that expresses the geometrical relationship is calculated, and this numerical matrix is defined as an absolute numerical matrix. The matrix calculation unit 12 is characterized when a numerical matrix cannot be calculated from the line segment or point information on each image notified from the input reception unit 16, or when the line segment or point information does not exist depending on the image. It is also possible to calculate a numerical matrix from a combination of feature points with little error that are extracted by the point extraction unit 11 and remain in the filtering unit 15. However, the numerical matrix calculated in this case is a numerical matrix similar to the conventional one, and is not an absolute numerical matrix.

Here, the process in the matrix calculation unit 12 will be described more specifically with reference to FIG. FIG. 4 is a diagram illustrating processing by the matrix calculation unit according to the embodiment of the present invention. In FIG. 4, E represents a epipolar plane, O _i denotes the camera center position of one image, O _'i denotes the center position of the camera of the other image. Further, in FIG. 4, the parallelograms shown on the left and right indicate the frame of the image, respectively.

Further, in the example of FIG. 4, a line segment L is designated in one image and a line segment L'is designated in the other image, and both line segments correspond to each other. In this case, in the line segment L, the epipolar line l _{n intersecting the start point and the epipolar line l n + m} intersecting at the end point are defined as absolute epipolar lines, and in the line segment L', the epipolar line l intersecting the start point is also defined. ' _{N and the epipolar line l'n + m} that intersects at the end point are defined as an absolute epipolar line.

Furthermore, the intersection x _i between the line segment L and the absolute epipolar line l _n, the _i 'intersection x of the _n' segment L and absolute epipolar line l, defined as a combination of absolute feature points. Further, the intersection x _j of the line segment L and the absolute epipolar line l _{n + m,} also the intersection x _'j' and absolute epipolar line l 'segment L and _{n + m,} is defined as a combination of absolute feature points.

As shown in FIG. 4, arbitrary epipolar lines l _{n + 1} and l _{n + 2} may be set _{between the epipolar lines l n} and the epipolar lines l _{n + m.} In this case, similarly, arbitrary epipolar lines l' _{n + 1} and l' _{n + 2} are set _{between the epipolar line l'n} and the epipolar line l' _{n + m.} Further, in this case, the intersection of the line segment L and the newly set epipolar lines l _{n + 1} and l _{n + 2} is the intersection of the line segment L'and the newly set epipolar lines l' _{n + 1} and l' _{n + 2,} respectively. It is a combination of absolute feature points. The interval between epipolar lines is set to an arbitrary value.

Therefore, in the example of FIG. 4, the matrix calculation unit 12 obtains an absolute combination of feature points as the geometrical relationship between the lines, and uses the obtained combination of feature points to form the relationship of the following number 1. From the equation, a Fundamental matrix (reference) is calculated as an absolute numerical matrix. In the following equation 1, "x" is a two-dimensional point obtained by projecting a point X in a three-dimensional space onto one of the images. "X'" is a two-dimensional point obtained by projecting a point X in a three-dimensional space onto the other image. T is a transposed matrix. Also, F is a Fundamental matrix. Further, in the present embodiment, the numerical matrix is not limited to the Fundamental matrix, and may be any matrix that can express a geometric relationship.

(Number 1)
x ^'T Fx = 0

References: Richard Hartley and Andrew Zisserman, “Multiple View Geometry in Computer Vision Second Edition”, Cambridge University Press, March 2004.

In the present embodiment, the shape building unit 13 first uses a numerical matrix (Fundamental matrix) calculated by the matrix calculation unit 12 to combine feature points extracted by the feature point extraction unit 11 (in the filtering unit 15). Exclude combinations of feature points with inconsistent geometric relationships from (excluding those excluded). After that, the shape building unit 13 constructs a three-dimensional shape using a combination of feature points from which the combination of feature points having inconsistent geometric relationships is excluded, and at that time, the input receiving unit 16 accepts the combination. , Corresponding points, or points on the corresponding line segments can also be used as a combination of feature points.

Further, the shape building unit 13 identifies a feature point that overlaps with a designated line segment or a point from a combination of feature points extracted by the feature point extraction unit 11 (excluding those excluded by the filtering unit 15). However, the combination of feature points including the specified feature points can also be excluded. This is because when the feature point corresponding to the feature point overlapping the one line segment or point is not on the other line segment or point, the combination of the feature points is likely to be incorrect. After that, the shape building unit 13 builds a three-dimensional shape by using the combination of the remaining feature points.

Here, the process in the shape building unit 13 will be described more specifically with reference to FIG. FIG. 5 is a diagram illustrating processing by the shape building unit according to the embodiment of the present invention. In FIG. 5, those with the reference numerals shown in FIG. 4 indicate those with the same symbols in FIG. Then, in FIG. 5, and whether it is whether a determination target is wrong combination of feature points p _i and the feature point p _'i.

In the example of FIG. 5, the shape construction unit 13 calculates the epipolar line l _'1 by using the relationships shown in Formula 2 below, corresponding to the epipolar lines l ₁ to the feature point p _i are present on the epipolar line do.

(Number 2)
l'= Fx

The shape building unit 13, when the calculated epipolar line l _'1 and the feature point p' the distance d between the _i may determine whether a threshold value or more, the distance d is equal to or greater than the threshold value, combination of feature point p _i and the feature point p _'i is judged to be wrong. In this case, the shape construction unit 13, with the exclusion of combination of the feature point p _i and the feature point p _'i, constructing a three-dimensional shape.

Further, in the present embodiment, the shape building unit 13 may temporarily construct a temporary three-dimensional shape by using a combination of feature points that have been filtered by the filtering unit 15. In this case, the user can identify a place where the restoration of the three-dimensional shape is insufficient based on the temporary three-dimensional shape, so that an appropriate line segment or point can be specified. After that, the shape building unit 13 reconstructs the three-dimensional shape.

Further, the display unit 17 displays the three-dimensional shape constructed by the shape construction unit 13 on the screen of the display device 20. Specifically, the display unit 17 creates image data for displaying the constructed three-dimensional shape on a two-dimensional screen, and outputs the created image data to the display device 20. Further, the display unit 17 displays the temporary three-dimensional shape on the screen of the display device 20 even when the shape construction unit 13 constructs the temporary three-dimensional shape.

[Device operation]
Next, the operation of the image processing apparatus 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flow chart showing the operation of the image processing device according to the embodiment of the present invention. In the following description, FIGS. 1 to 5 will be referred to as appropriate. Further, in the present embodiment, the image processing method is implemented by operating the image processing device 10. Therefore, the description of the image processing method in the present embodiment is replaced with the following description of the operation of the image processing device 10.

As shown in FIG. 6, first, the image acquisition unit 14 acquires image data of each of a plurality of images showing a three-dimensional shape construction target from an external device (step A1).

Next, the feature point extraction unit 11 calculates the feature amount for each image of the image data acquired in step A1, identifies the feature points, and sets the corresponding feature points between the images to each other. Extract as a combination (step A2).

Next, the filtering unit 15 calculates the geometrical relationship between the feature points for each combination of the feature points extracted in step A2, and identifies an erroneous feature point combination based on the calculation result. The combination of the specified feature points is excluded (step A3).

Next, the shape building unit 13 builds a temporary three-dimensional shape by using the combination of the feature points after the filtering process in step A3 is performed (step A4). Further, when step A4 is executed, the display unit 17 displays the temporary three-dimensional shape constructed in step A4 on the screen of the display device 20.

When step A4 is executed and the tentative three-dimensional shape is displayed on the screen of the display device 20, the user can use a line segment corresponding to each other on each image via an input device (not shown in FIG. 2). Or specify a point. As a result, the input receiving unit 16 receives the input of the designated line segment or point (step A5). Further, the input receiving unit 16 notifies the matrix calculation unit 12 of the information of each designated line segment or the information of each point.

Next, when step A5 is executed, the matrix calculation unit 12 identifies the geometric relationship between the line segment or the point based on the information of each line segment or the information of each point, and the specified geometric relationship. Is calculated (step A6).

Next, the shape building unit 13 uses the numerical matrix calculated in step A6 to exclude the combination of feature points having inconsistent geometric relationships from the combination of feature points after the application of the process of step A3 (). Step A7).

Next, the shape building unit 13 again builds the three-dimensional shape using the combination of the remaining feature points (step A8). After that, the display unit 17 displays the three-dimensional shape constructed in step A8 on the screen of the display device 20 (step A9).

[Effect of embodiment]
As described above, according to the present embodiment, it is possible to construct a three-dimensional shape by eliminating a combination of wrong feature points that are not on the same scanning line. Therefore, according to the present embodiment, when extracting the corresponding combination of feature points from a plurality of images, it is possible to suppress the extraction of the wrong combination of feature points. As a result, the accuracy of the restored three-dimensional shape is improved.

[program]
The program in the present embodiment may be any program as long as it causes a computer to execute steps A1 to A9 shown in FIG. By installing and executing this program on a computer, the image processing device 10 and the image processing method according to the present embodiment can be realized. In this case, the computer processor functions as a feature point extraction unit 11, a matrix calculation unit 12, a shape construction unit 13, an image acquisition unit 14, a filtering unit 15, an input reception unit 16, and a display unit 17 to perform processing.

Further, the program in the present embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may be used as one of a feature point extraction unit 11, a matrix calculation unit 12, a shape construction unit 13, an image acquisition unit 14, a filtering unit 15, an input reception unit 16, and a display unit 17, respectively. It may work.

Here, a computer that realizes the image processing apparatus 10 by executing the program according to the present embodiment will be described with reference to FIG. 7. FIG. 7 is a block diagram showing an example of a computer that realizes the image processing apparatus according to the embodiment of the present invention.

As shown in FIG. 7, the computer 110 includes a CPU 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. Each of these parts is connected to each other via a bus 121 so as to be capable of data communication. The computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to the CPU 111 or in place of the CPU 111.

The CPU 111 expands the programs (codes) of the present embodiment stored in the storage device 113 into the main memory 112 and executes them in a predetermined order to perform various operations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Further, the program in the present embodiment is provided in a state of being stored in a computer-readable recording medium 120. The program in the present embodiment may be distributed on the Internet connected via the communication interface 117.

Further, specific examples of the storage device 113 include a semiconductor storage device such as a flash memory in addition to a hard disk drive. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

The data reader / writer 116 mediates the data transmission between the CPU 111 and the recording medium 120, reads the program from the recording medium 120, and writes the processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

Specific examples of the recording medium 120 include a general-purpose semiconductor storage device such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), a magnetic recording medium such as a flexible disk, or a CD-. Examples include optical recording media such as ROM (Compact Disk Read Only Memory).

The image processing device 10 in the present embodiment can also be realized by using the hardware corresponding to each part instead of the computer in which the program is installed. Further, the image processing apparatus 10 may be partially realized by a program and the rest may be realized by hardware.

A part or all of the above-described embodiments can be expressed by the following descriptions (Appendix 1) to (Appendix 12), but the present invention is not limited to the following description.

(Appendix 1)
A device for constructing a three-dimensional shape from multiple images.
A feature point extraction unit that extracts a combination of corresponding feature points from each of the plurality of images,
When two or more line segments or two or more points corresponding to each other are specified on at least two of the plurality of images, the geometrical relationship between the two or more line segments corresponding to each other, or each other. A matrix calculation unit that identifies a geometric relationship between two or more corresponding points and calculates a numerical matrix that expresses the specified geometric relationship.
Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are specified, and the feature points other than the specified combinations of the feature points are specified. The shape building unit that builds the three-dimensional shape using the combination of
An image processing device characterized by being equipped with.

(Appendix 2)
Among the feature points constituting the combination of the extracted feature points, the shape building unit identifies and identifies the feature points that overlap with two or more designated line segments or two or more points corresponding to each other. The three-dimensional shape is constructed by eliminating the combination of the feature points including the feature points.
The image processing apparatus according to Appendix 1.

(Appendix 3)
It further includes a display unit that displays the three-dimensional shape constructed by the shape construction unit on the screen.
The image processing apparatus according to Appendix 1 or 2.

(Appendix 4)
The shape building unit constructs a temporary three-dimensional shape by using the combination of the feature points extracted by the feature point extracting unit.
The display unit further displays the temporary three-dimensional shape on the screen.
The image processing apparatus according to Appendix 3.

(Appendix 5)
A method for constructing a 3D shape from multiple images,
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. , Or a step that identifies a geometric relationship between two or more points that correspond to each other and calculates a numerical matrix that expresses the identified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are specified, and the combinations of the feature points other than the specified combinations are specified. A step of constructing the three-dimensional shape using the combination of the feature points, and
An image processing method comprising.

(Appendix 6)
In step (c) above,
Among the feature points constituting the combination of the feature points extracted in the step (a), the feature points overlapping the designated two or more line segments or two or more points corresponding to each other are specified and specified. The three-dimensional shape is constructed by eliminating the combination of the feature points including the feature points.
The image processing method according to Appendix 5.

(Appendix 7)
(D) Further having a step of displaying the three-dimensional shape constructed by the step of (c) on the screen.
The image processing method according to Appendix 5 or 6.

(Appendix 8)
(E) A step of constructing a temporary three-dimensional shape using the combination of the feature points extracted by the step (a), and
(F) A step for displaying the temporary three-dimensional shape constructed in the step (e) on the screen, and
The image processing method according to Appendix 7, further comprising.

(Appendix 9)
The computer, a program for constructing a three-dimensional shape from a plurality of images,
To the computer
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) When two or more line segments or two or more points corresponding to each other are designated on at least two of the plurality of images, the geometric relationship between the two or more line segments corresponding to each other is specified. , Or a step that identifies a geometric relationship between two or more points that correspond to each other and calculates a numerical matrix that expresses the identified geometric relationship.
(C) Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are specified, and the combinations of the feature points other than the specified combinations are specified. A step of constructing the three-dimensional shape using the combination of the feature points, and
Including Mpu Rogura-time instructions to the execution.

(Appendix 10)
In step (c) above,
Among the feature points constituting the combination of the feature points extracted in the step (a), the feature points overlapping the designated two or more line segments or two or more points corresponding to each other are specified and specified. The three-dimensional shape is constructed by eliminating the combination of the feature points including the feature points.
The program described in Appendix 9.

(Appendix 11)
The program is on the computer
(D) Further including an instruction to execute the step, displaying the three-dimensional shape constructed by the step (c) on the screen.
The program according to Appendix 9 or 10.

(Appendix 12)
The program is on the computer
(E) A step of constructing a temporary three-dimensional shape using the combination of the feature points extracted by the step (a), and
(F) A step for displaying the temporary three-dimensional shape constructed in the step (e) on the screen, and
Including further instructions to execute,
The program described in Appendix 11.

Although the invention of the present application has been described above with reference to the embodiments, the invention of the present application is not limited to the above-described embodiments. Various changes that can be understood by those skilled in the art can be made within the scope of the present invention in terms of the configuration and details of the present invention.

This application claims priority on the basis of Japanese application Japanese Patent Application No. 2017-190377 filed on September 29, 2017 and incorporates all of its disclosures herein.

As described above, according to the present invention, when extracting the combination of corresponding feature points from a plurality of images, it is possible to suppress the extraction of the wrong combination of feature points. The present invention is useful in a technique for constructing a three-dimensional shape from a plurality of images such as SfM.

10 Image processing device 11 Feature point extraction unit 12 Matrix calculation unit 13 Shape construction unit 14 Image acquisition unit 15 Filtering unit 16 Input reception unit 17 Display unit 110 Computer 111 CPU
112 Main memory 113 Storage device 114 Input interface 115 Display controller 116 Data reader / writer 117 Communication interface 118 Input device 119 Display device 120 Recording medium 121 Bus

Claims (12)

A device for constructing a three-dimensional shape from multiple images.
A feature point extraction unit that extracts a combination of corresponding feature points from each of the plurality of images,
For each combination of the extracted feature points, the geometric relationship between the feature points is calculated, the wrong feature point combination is specified based on the calculation result, and the specified wrong feature point combination is specified. Exclude, filtering part,
When two or more independent line segments or two or more points different from the extracted feature points are designated on at least two of the plurality of images, two or more points corresponding to each other are specified. A matrix calculation unit that specifies the geometrical relationship between the line segments of the above, or the geometrical relationship between two or more points corresponding to each other, and calculates a numerical matrix expressing the specified geometrical relationship.
Using the numerical matrix, among the extracted combinations of the feature points, the combinations of the feature points in which the geometrical relationships between the feature points are inconsistent are specified, and the feature points other than the specified combinations of the feature points are specified. The shape building unit that builds the three-dimensional shape using the combination of
Equipped with
Further, even when the number of combinations of the feature points is insufficient and the numerical matrix cannot be calculated, the matrix calculation unit is independent from the feature points and has two or more line segments or two or more points corresponding to each other. By specifying, a numerical matrix is calculated from the combination of feature points with few errors remaining by the processing of the filtering unit.
An image processing device characterized by this. Among the feature points constituting the combination of the extracted feature points, the shape building unit identifies and identifies the feature points that overlap with two or more designated line segments or two or more points corresponding to each other. The three-dimensional shape is constructed by eliminating the combination of the feature points including the feature points.
The image processing apparatus according to claim 1. It further includes a display unit that displays the three-dimensional shape constructed by the shape construction unit on the screen.
The image processing apparatus according to claim 1 or 2. The shape building unit constructs a temporary three-dimensional shape by using the combination of the feature points extracted by the feature point extracting unit.
The display unit further displays the temporary three-dimensional shape on the screen.
The image processing apparatus according to claim 3. A method for constructing a 3D shape from multiple images,
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) For each combination of the extracted feature points, the geometric relationship between the feature points is calculated, and based on the calculation result, the wrong combination of the feature points is specified, and the specified wrong feature point is specified. Eliminate the combination of steps and
(C) When two or more independent line segments or two or more points different from the extracted feature points are designated on at least two of the plurality of images, they correspond to each other. A step of identifying a geometric relationship between two or more line segments, or a geometric relationship between two or more points corresponding to each other, and calculating a numerical matrix expressing the specified geometric relationship.
(D) Among the extracted combinations of the feature points, the combination of the feature points whose geometrical relationship between the feature points is inconsistent is specified by using the numerical matrix, and the combination of the feature points other than the specified combination of the feature points is specified. A step of constructing the three-dimensional shape using the combination of the feature points, and
Have a,
Further, in the step (c), even when the number of combinations of the feature points is insufficient and the numerical matrix cannot be calculated, two or more independent line segments different from the feature points or two or more corresponding to each other or two or more. By designating the points, the numerical matrix is calculated from the combination of the feature points with less error left by the step (b) .
An image processing method characterized by that. In step (d) above,
Among the feature points constituting the combination of the feature points extracted in the step (a), the feature points overlapping the designated two or more line segments or two or more points corresponding to each other are specified and specified. The three-dimensional shape is constructed by eliminating the combination of the feature points including the feature points.
The image processing method according to claim 5. (E) Further having a step of displaying the three-dimensional shape constructed by the step (d) on the screen.
The image processing method according to claim 5 or 6. (F) A step of constructing a temporary three-dimensional shape using the combination of the feature points extracted by the step (a), and
(G) A step for displaying the temporary three-dimensional shape constructed in the step (f) on the screen, and
The image processing method according to claim 7, further comprising. The computer, a program for constructing a three-dimensional shape from a plurality of images,
To the computer
(A) A step of extracting a combination of corresponding feature points from each of the plurality of images.
(B) For each combination of the extracted feature points, the geometric relationship between the feature points is calculated, and based on the calculation result, the wrong combination of the feature points is specified, and the specified wrong feature point is specified. Eliminate the combination of steps and
(C) When two or more independent line segments or two or more points different from the extracted feature points are designated on at least two of the plurality of images, they correspond to each other. A step of identifying a geometric relationship between two or more line segments, or a geometric relationship between two or more points corresponding to each other, and calculating a numerical matrix expressing the specified geometric relationship.
(D) Among the extracted combinations of the feature points, the combination of the feature points whose geometrical relationship between the feature points is inconsistent is specified by using the numerical matrix, and the combination of the feature points other than the specified combination of the feature points is specified. A step of constructing the three-dimensional shape using the combination of the feature points, and
Have a,
Further, in the step (c), even when the number of combinations of the feature points is insufficient and the numerical matrix cannot be calculated, two or more independent line segments different from the feature points or two or more corresponding to each other or two or more. By designating the points, the numerical matrix is calculated from the combination of the feature points with less error left by the step (b) .
A program that contains instructions to execute. In step (d) above,
Among the feature points constituting the combination of the feature points extracted in the step (a), the feature points overlapping the designated two or more line segments or two or more points corresponding to each other are specified and specified. The three-dimensional shape is constructed by eliminating the combination of the feature points including the feature points.
The program according to claim 9. The program is on the computer
(E) Further including an instruction to execute the step, displaying the three-dimensional shape constructed by the step (d) on the screen.
The program according to claim 9 or 10. The program is on the computer
(F) A step of constructing a temporary three-dimensional shape using the combination of the feature points extracted by the step (a), and
(G) A step for displaying the temporary three-dimensional shape constructed in the step (f) on the screen, and
Including further instructions to execute,
The program according to claim 11.

Images