JP7242431B2 - Three-dimensional measuring device, three-dimensional measuring method and three-dimensional measuring program - Google Patents

Description

The present invention relates to three-dimensional measurement technology using images.

A technique is known for acquiring data relating to the three-dimensional shape of a plant using an image and three-dimensionally ascertaining the growth process of the plant. For example, Patent Literature 1 describes a technique of photographing a plant, which is an object to be measured, while rotating the plant, and obtaining a three-dimensional model of the subject (plant) from the obtained multiple photographed images.

JP 2018-197685 A

In three-dimensional measurement using images, matching (identification of correspondence) between multiple images obtained from different viewpoints is important. FIG. 1(A) shows a case where an object to be measured (a flower in this case) placed on a turntable is photographed by a camera while being rotated, and three-dimensional measurement of the object to be measured is performed. In this case, considering the coordinates fixed to the turntable (object to be measured), the object to be measured is photographed from a plurality of positions around it. In this technique, the position of each camera is specified by calculation by the posterior resection method, and the three-dimensional position of each part of the measurement object is calculated by the anterior resection method.

Ideally, as shown in FIG. 1(B), the relative positions of each camera with respect to the turntable are distributed on the circumference. However, in practice, as shown in FIG. 2, a deviation occurs between the camera position (1) at the start of calculation and the camera position (16) calculated last. This is because errors gradually accumulate in the process of calculating the camera position in order of (1)→(2)→(3)→(4)→ . . .

This deviation is corrected by performing the bundle adjustment calculation after the camera position calculation. However, the bundle adjustment calculation at the stage where the error becomes large as shown in FIG. 2 imposes a heavy load on the processor, and the calculation may not converge.

Against this background, the present invention provides a technique for suppressing deterioration in the calculation accuracy of the camera position in a technique for performing three-dimensional photo measurement of a measurement target by photographing the measurement target from a plurality of surrounding viewpoints. With the goal.

The present invention provides an image data reception unit that receives a plurality of image data obtained by photographing an object to be measured and a random dot pattern from a plurality of surrounding viewpoints with a camera, and a plurality of the cameras that photograph the plurality of images. A camera position selection unit that selects three or more camera positions surrounding the measurement object from among positions, and a stereo image that selects a stereo image from images captured from the camera positions selected by the camera position selection unit. a selection unit and a camera position calculation unit that calculates the position of the camera that captured the stereo image by a resection method using the stereo image, and the selection of the camera position is performed a plurality of times, and different selections are made. Sometimes, at least some of the selected camera positions are non-overlapping 3D measurement devices.

Further, the present invention provides an image data reception unit that receives a plurality of image data obtained by photographing an object to be measured and a random dot pattern from a plurality of surrounding viewpoints with a camera, and a plurality of the cameras photographing the plurality of images. A camera position selection unit that selects three or more camera positions surrounding the measurement object from among the positions of the camera position selection unit, and a stereo that selects a stereo image from images captured from the camera positions selected by the camera position selection unit and a camera position calculation unit that calculates the position of the camera that captured the stereo image by a resection method using the stereo image, and the selection of the camera position is performed a plurality of times and is different. A three-dimensional measurement apparatus in which camera positions to be selected do not overlap at the time of selection.

In the present invention, the camera position selection unit selects N camera positions at angular positions of 360°/N (N is a natural number of 3 or more) as viewed from the object to be measured. , the second selection of selecting N camera positions from a position where the angular position seen from the measurement object is shifted by 180°/N from the N camera positions selected in the first time; The mode performed is mentioned.

In the present invention, there is a mode in which the random dot pattern is formed on the surface of a longitudinal member fixed to a stand on which the measurement object is placed and extending upward from the stand. In this aspect, it is preferable that the dimension in the height direction of the longitudinal member is larger than the dimension in the height direction of the object to be measured placed on the table. Further, there is an aspect in which the table is rotatable.

In the present invention, the object to be measured and the random dot pattern are photographed from a plurality of surrounding viewpoints by photographing the object to be measured by the camera while rotating the table with respect to the camera. For example, a plurality of image data are obtained, and the center of rotation of the object to be measured is positioned within a polygon whose vertices are three or more positions surrounding the object to be measured.

The present invention comprises an image data receiving step of receiving a plurality of image data obtained by photographing an object to be measured and a random dot pattern from a plurality of surrounding viewpoints with a camera; A camera position selection step of selecting three or more camera positions surrounding the object to be measured from among positions, and a stereo image of selecting a stereo image from images captured from the camera positions selected in the camera position selection step. and a camera position calculation step of calculating the position of the camera that captured the stereo image by a resection method using the stereo image, wherein the selection of the camera position is performed a plurality of times and different selections are made. Sometimes, it can also be understood as a three-dimensional measurement method in which at least some selected camera positions do not overlap.

The present invention is a program to be read and executed by a computer, and includes an image data receiving step for receiving a plurality of image data obtained by photographing an object to be measured and a random dot pattern with a camera from a plurality of surrounding viewpoints. a camera position selection step of selecting three or more camera positions surrounding the measurement object from among the positions of the cameras that captured the plurality of images; and the camera positions selected in the camera position selection step. Execute a stereo image selection step of selecting a stereo image from images taken from and a camera position calculation step of calculating the position of the camera that took the stereo image by the resection method using the stereo image, The selection of the camera positions is performed a plurality of times, and it can also be understood as a three-dimensional measurement program in which at least some of the selected camera positions do not overlap when different selections are made.

According to the present invention, in the technique of performing three-dimensional photo measurement of a measurement object by photographing the measurement object from a plurality of surrounding viewpoints, it is possible to obtain a technique that suppresses deterioration in the calculation accuracy of the camera position.

1A and 1B are principle diagrams for explaining the principle of three-dimensional measurement; FIG. It is a principle diagram explaining the principle which an error arises. 1 is a principle diagram of an example of the invention; FIG. 1 is a principle diagram of an example of the invention; FIG. 1 is a principle diagram of an example of the invention; FIG. 1 is a principle diagram of an example of the invention; FIG. 1 is a conceptual diagram of an embodiment; FIG. 1 is a block diagram of a three-dimensional measuring device according to an embodiment; FIG. It is a principle diagram (A) showing the principle of the posterior resection method and a principle diagram (B) showing the principle of the anterior resection method. 4 is a flow chart showing an example of a procedure of processing; 4 is a flow chart showing an example of a procedure of processing; The figure which shows the camera position in an up-down direction.

1. Principle FIG. 1 shows the basic principle of the present invention. In Fig. 1(A), the object to be measured (a flower in this case) is photographed by the camera while the turntable is rotated with respect to a fixed camera. When to do so is shown. Here, considering the coordinate system fixed to the turntable, the camera position (viewpoint position) with respect to the turntable is distributed on the circumference centered on the rotation center of the turntable, as shown in FIG. 1(B). . It should be noted that FIG. 1B is drawn from the viewpoint of viewing the turntable from above along the rotation axis of the turntable. This also applies to FIGS. 2 to 6. FIG.

Here, when the camera position is calculated in the order of (1)→(2)→(3)→... using the resection method, errors accumulate as the calculation is repeated, and as shown in FIG. The calculated camera position deviates from the actual position.

In order to avoid the accumulation of this deviation, the present invention adopts a method shown in FIG. 3 as an example. In this method, for example, using four images taken from four equiangular positions (1), (9), (17), and (25) at 90° intervals surrounding the turntable, Calculate the camera position.

Calculation of the camera position includes selection of stereo images, extraction of feature points from the stereo images, detection of targets, and three-dimensional positional relationships (three-dimensional model ) (relative orientation), and assigning a scale to the three-dimensional model using known dimensions and known target positions (absolute orientation).

Stereo images are selected from pairs taken from two camera positions separated by 90° at adjacent angular positions such as (1) and (9), (9) and (17), and (17) and (25). be done. After calculating the camera positions at the four positions (1), (9), (17), and (25), the bundle adjustment calculation is performed to minimize the error.

After calculating the four camera positions shown in FIG. 3, the process proceeds to FIG. In FIG. 4, in addition to the camera positions (1), (9), (17), and (25) obtained in the previous stage of FIG. 3, four positions rotated by 45° from these camera positions The same processing as in the case of FIG. 3 is performed using eight images taken from eight points including camera positions (5), (13), (21), and (29).

At this time, the four camera positions (1), (9), (17), and (25) in FIG. 5), (13), (21), and (29) are calculated. At this time, the camera positions (1), (9), (17), and (25) in FIG. 5 are initial values, but they are not determined values. It is recalculated so that the whole is optimized. In this way, the camera position with respect to the turntable is obtained at eight equally spaced points (at 45° intervals) on the circumference centered on the turntable. At this stage, the bundle calculation is performed again to minimize the error.

After obtaining the camera positions in FIG. 4, the camera positions (5), (13), (21), and (29) are rotated by 22.5° and similar processing is performed. Thus, 12 camera positions are obtained. This process is repeated until all camera positions are obtained. In the method shown in FIGS. 3 and 4, errors in the calculations are not accumulated or are small. Therefore, as shown in FIG. 2, the problem that the error gradually increases as the calculation progresses is suppressed.

In this method, first, all camera positions are obtained gradually and densely in the order of 4 directions -> 8 directions -> . At the same time, corresponding points are obtained by sparse stereo matching. At this time, error minimization (bundle adjustment) is also performed.

As a result, the camera position is obtained using all images, but the initial value is an image with a long baseline (in this example, the position is different in angle position by 90 degrees), and the whole is a closed loop, so the accuracy is good. . In addition, since the bundle adjustment at each stage is applied to the initial value, convergence does not slow down or fail (because there is an initial value that is correct, the convergence is quick).

For the point cloud of the object to be measured, after the camera position is determined, a dense three-dimensional point cloud is determined again by adjacent stereo matching. At that time, the sparse point group obtained when specifying the camera position is also used. Since the camera position has already been obtained here, bundle adjustment is not performed (of course, it may be performed). Note that it is also possible to perform alignment by the ICP method instead of bundle adjustment.

The number of camera positions surrounding the turntable (object to be measured) must be three or more. “Positions surrounding the turntable” refer to the position of the turntable when viewed from the direction of the rotation axis of the turntable (vertically above) and forming a polygon with three or more camera positions (viewpoint positions) as vertices. is defined as the state where the center of rotation of is located inside the polygon.

The shift angle θ at the M-th shift in the method of FIGS. 3 and 4 is given by θ=(360°/(N×(M+1)), where N is the number of equiangular angular positions of 3 or more. For example, in Fig. 4, M = 1 and N = 4, so θ = 45° The value of θ is not strict, and the camera is positioned at the position obtained by the above formula. If there is no camera (which may be possible depending on the shooting timing), a nearby camera position should be selected.

The selection of the camera position in the method of FIGS. 3 and 4 is performed based on data on the angular position of the turntable and the imaging timing. The angular position of the turntable is obtained from a sensor such as a rotary encoder that detects the rotation of the turntable, and the shooting timing is obtained from log data that stores the output timing of the control signal that controls the shooting timing of the camera.

As images taken from different camera positions that constitute a stereo image, two images taken from two camera positions are basically used, but three or more images taken from three or more different viewpoints are used. is also possible. If the requested angular position of the camera deviates from the actual camera position, the actual camera position closest to the requested camera position is adopted.

2. Overall Configuration FIG. 7 shows a three-dimensional measurement system 100 for obtaining three-dimensional data of a plant 120. As shown in FIG. The three-dimensional measurement system 100 includes a base 111 and a turntable 112 that rotates on the base by driving means such as a motor. A plant 122 , which is an object to be measured for three-dimensional measurement, is placed on the turntable 112 .

The base 111, the turntable 112, and the background of the three-dimensional measurement system 100 are colored in a color different from the color of the plant 120 (blue in this example). This color is preferably a monotone color different from the color of the plant 120 . For example, if the plant 120 is greenish, monotone colors different from the plant 120 such as blue, yellow, black, and white are selected. The object 120 is planted in a pot 124, and the pot 124 displays a two-dimensional barcode display 125 for identification.

The dimensions of each part of the base 111 and the turntable 112 are obtained in advance as known data. A target 113 serving as a reference point is displayed on the upper surface of the turntable 112 . Three or more targets 113 are arranged. The target 113 is a coded target with a two-dimensional code, and each can be individually identified. In addition, the position of each target on the turntable 112 is checked in advance and is known.

Random dot columns 121 to 123, which are identification members for image matching, are fixed on the upper surface of the turntable 112. As shown in FIG. The random dot columns 121 to 123 are plate-like members having a longitudinal shape, extend vertically upward from the turntable 112, and have random dot patterns displayed on their surfaces. In addition to the random dot pattern, identification codes are displayed on the surfaces of the random dot columns 121 to 123, and each identification code can be individually identified. A cylinder or a polygonal prism (for example, a square prism or a pentagonal prism) can also be used as the random dot column.

The number of random dot columns may be four or more. However, since the random dot columns are positioned between the cameras 131 to 133 and the plant 122, which is the object to be measured, if the number of random dot columns is too large, the cameras 131 to 133 will not be able to photograph the plant 122.

The positional relationship of the random dot columns 121-123 with respect to the turntable 112 is known, and the dimensions (width, length, thickness) of the random dot columns 121-123 are also known. Since the dimensions of the random dot columns 121-123 are known, the random dot columns 121-123 are also used as scales. A method of arranging a ruler on the turntable 112 and a mode of displaying a scale on the turntable 112 are also possible. The longitudinal dimension (vertical dimension) of the random dot columns 121 to 123 is set so that the tips of the random dot columns 121 to 123 are positioned above the upper end of the plant 120, which is the object to be measured. there is In addition, the random dot columns 121-123 are arranged so that the measurement object (the plant 120) is surrounded by the random dot columns 121-123 when viewed from above. This is the same when the number of random dot columns is increased.

A random dot pattern is a dot pattern in which circular dots are randomly distributed without regularity. Techniques related to random dot patterns are disclosed, for example, in Japanese Patent Application Laid-Open Nos. 11-39505, 10-97641, and Institute of Electronics, Information and Communication Engineers Research Report. COMP, Combation 112(24), 51-58, 2012-05-07, etc. The color of the random dot pattern may be monotone or multicolor. In this example, a black random dot pattern is displayed on a white background.

Cameras 131 to 133 are arranged around the turntable 112 . The cameras 131 to 133 are fixed to the base 111 and repeatedly photograph the rotating turntable 112 and the plant 120 thereon at specific time intervals. At least one camera is sufficient, but it is desirable to arrange a plurality of cameras and set the number and positions of the cameras so as not to produce blind spots with respect to the plant 120 as much as possible. The cameras 131 to 133 are digital cameras, and data of captured images (image data) are sent to the three-dimensional measuring device 200 .

The plant 120 is continuously photographed from the cameras 131 to 133 fixed to the base 111 while rotating the turntable 112 . As a result, a large number of images of the plant 120 captured from a large number of surrounding camera positions (viewpoints) are obtained. At this time, the photographing ranges of images from adjacent camera positions are partially overlapped.

When the above photographing is performed, a plurality of photographed images obtained by photographing the surroundings of the plant 120, which is the object to be measured, are obtained. Here, images taken from adjacent viewpoints (camera positions) are partially overlapped. For example, consider a case where 72 images are taken from one camera facing the plant 120 while the turntable 112 rotates once. In this case, considering the coordinate system fixed to the turntable 112, the position of the viewpoint (camera position) is gradually shifted in increments of 360°/72=5° on the circumference around the center of rotation of the turntable 112. 72 images photographed with shifting are obtained.

3. Three-Dimensional Measuring Apparatus A three-dimensional model of the plant 120, which is an object to be measured, is created based on the known principle of three-dimensional photogrammetry. The three-dimensional measuring device 200 is configured using a general-purpose PC (personal computer). The three-dimensional measuring device 200 creates a three-dimensional model of the plant 120 based on image data captured by the cameras 131-133. The three-dimensional measuring device 200 also sends a control signal to the motor driving device 140 to control the rotation of the turntable 112 . The three-dimensional measurement apparatus 200 also controls the imaging timing of the cameras 131-133.

The three-dimensional measuring device 200 is a computer equipped with a CPU, a memory device, and various interfaces, and can be configured with general-purpose or dedicated hardware. General-purpose hardware constituting the three-dimensional measuring apparatus 200 includes a WS (work station) in addition to the PC. An operation program that realizes the functions of the three-dimensional measuring apparatus 200 is installed in these computers, and the functions of each functional unit described later are realized by software.

Part or all of the three-dimensional measuring device 200 may be configured by a dedicated arithmetic circuit. Also, a functional unit configured by software and a functional unit configured by a dedicated arithmetic circuit may be combined.

For example, each functional unit shown in the figure is composed of electronic circuits and processors such as PLDs (Programmable Logic Devices) represented by CPUs (Central Processing Units), ASICs (Application Specific Integrated Circuits), and FPGAs (Field Programmable Gate Arrays). be. It is also possible to configure part of the functions with dedicated hardware and configure the other part with a general-purpose microcomputer.

It is also possible to use a PC, tablet, smart phone, or the like connected to the server via the Internet as an operation interface terminal, with the functions of the three-dimensional measuring apparatus 200 executed by a server.

The three-dimensional measuring apparatus 200 includes an image data reception unit 210, a camera position selection unit 220, a stereo image selection unit 221, a target detection unit 240, an exterior orientation element calculation unit 245, a feature point extraction unit 250, a corresponding point identification unit 260, a tertiary An original coordinate calculation unit 270 , a three-dimensional model creation unit 280 , a storage unit 290 , a rotation control unit 295 , an imaging timing control unit 296 and a bundle adjustment calculation unit 297 are provided. The three-dimensional measuring apparatus 200 also has an interface function for exchanging data with an external device and a user interface function for receiving operations by an operator. These use the interface of the PC to be used.

The image data reception unit 210 receives image data (image data) captured by the cameras 131 to 133 and imports the data into the three-dimensional measuring apparatus 100 .

The camera position selection unit 220 selects candidate camera positions calculated by the resection method as illustrated in FIG. When the camera position is selected, it is in a state before the camera position is calculated by the resection method, and the exact coordinates of the camera position are unknown. However, it is possible to specify the approximate positions of the cameras 131 to 133 with respect to the turntable 112 and select the images shot there from the content of the rotation control of the turntable 112 and the control of the imaging timings of the cameras 131 to 133 . This processing is performed by the camera position selection unit 220 .

For example, since the rotational position of the turntable 112 is detected by a rotary encoder or the like, it is possible to know the rotational position of the turntable 112 on the time axis. The position of the camera at the time of photographing can be known from this information and the data of photographing time. For example, the camera position (1) and the camera position (9) can be identified. However, this camera position is an approximate position, and its accuracy is not at a level that can be used for three-dimensional photogrammetry.

The stereo image selection unit 221 selects two or more images obtained by photographing overlapping locations from different viewpoints as stereo images. For example, an image of the turntable 112 photographed from the positions (1) and (5) in FIG. 5 and an image of the turntable and the plant 120 thereon from (1) and (17) in FIG. 6 are stereo images. is selected as A stereo image is basically a set of two images, but three or more images can be used as a set to form a stereo image. The image should include at least the turntable 112, the plant 120, which is the object to be measured, and at least one of the random dot columns 121-123.

The target detection unit 240 detects the target 113 appearing in the images captured by the cameras 131-133. An image of the target 113 is obtained in advance, and is used as a reference to extract the target 113 from the photographed image. Since the target 113 can be identified and the position on the turntable 112 is known, the target detection unit 240 detects the position of each of the multiple targets 113 .

The exterior orientation element calculation unit 245 calculates the exterior orientation elements (position and orientation) of the cameras 131 to 133 in FIG. Calculated by the method. FIG. 9(A) shows the principle of the posterior resection method. The retrosection method is a method of observing directions from an unknown point to three or more known points and determining the position of the unknown point as the intersection of those direction lines. Examples of the retrosection method include single photo orientation and DLT method (Direct Liner Transformation Method). The resection method is described in Basic Surveying (Denkishoin: 2010/4 issue) p.182, p.184. Further, Japanese Patent Application Laid-Open No. 2013-186816 discloses an example of a specific calculation method regarding the resection method.

An example of specific calculation will be described below. Here, it is assumed that the turntable 112 is rotated and the turntable 112 is photographed by the camera 132 at two different timings, before and after the turntable 112 is rotated by 30°. Rotation control section 295 controls 30° rotation, and imaging timing control section 296 controls imaging timing. In this example, the rotation of the turntable 112 and the timing of photographing by the camera 132 are controlled so that the rotation is continued at a constant speed and photographing is performed at angular positions different by 30° when viewed from above.

In this case, if the coordinate system fixed to the turntable 112 is used, images taken from positions (1) and (3) in FIG. 5 can be obtained. These two photographed images are used as stereo images, feature points are extracted therefrom, and matching (identification of correspondence relationship) between the extracted feature points between the stereo images is performed. A feature point extraction unit 250 extracts feature points, and a corresponding point specification unit 260 specifies corresponding points.

After specifying the correspondence relationship of the feature points between the stereo images taken from camera positions (1) and (3) in FIG. ) is calculated. Here, the camera position is described in a local coordinate system fixed to the turntable 112 with the rotation center of the turntable 112 as the origin.

For example, the exterior orientation parameters (attitude and position) of the camera at camera positions (1) and (3) in FIG. 5 are obtained by the following method. First, the upper surface of the turntable 112 is a plane, and the feature points acquired therefrom are located on the plane. Random dots on the surfaces of the random dot columns 121 to 123 are distributed on the vertical plane. From this constraint condition, the position and orientation of the camera positions (1) and (3) in the relative 3D model can be obtained by relative orientation.

For example, P ₁ , P ₂ , and P ₃ in FIG. 9A are selected from the feature points obtained from the dots of the corresponding random dot columns 121 to 123 in the stereo image. Here, a constraint is imposed that P ₁ , P ₂ and P ₃ are distributed on the vertical plane. When the positions of P ₁ , P ₂ , and P ₃ in the screen are p ₁ , p ₂ , and p ₃ , three points connecting P ₁ and p ₁ , P ₂ and p ₂ , and P ₃ and p ₃ Set direction lines. In this case, the intersection O of the three direction lines is the position (projection center) of the camera 132 that captured the image. Also, the extension direction of the direction line connecting the point O and the center of the screen is the optical axis direction of the camera 132, and the orientation (orientation) of the camera 132 is obtained.

Since the absolute positional relationship of P ₁ , P ₂ and P ₃ is unknown at this stage, the three-dimensional model of FIG. 9A in this case is a relative three-dimensional model. Relative orientation is the process of obtaining this relative three-dimensional model.

Here, if _P1 and _P2 are the endpoints of the upper side of the random dot column 121, the coordinates of those endpoints in the local coordinate system fixed to the turntable 112 are known, so the relative three-dimensional model has a scale. Given. An absolute orientation is thus obtained, and the exterior orientation parameters of the camera in that coordinate system are known. In this method, positional information of multiple targets 113 can also be used.

It is also possible to obtain exterior orientation parameters of the camera at each camera position from the position information of three or more targets 113 . For example, assume that three targets 113 are visible from position (1) in FIG. In this case, if P ₁ , P ₂ and P ₃ are the positions of these three targets 113, the coordinates of point O can be obtained since the positions of P ₁ , P ₂ and P ₃ are known. Also, the orientation of the camera can be determined from the direction connecting the point O and the center of the screen. Thus, the exterior orientation parameters of the camera at position (1) where three targets 113 are visible are known.

Also, if the camera position is known, the positions of many feature points whose positions are not known in the coordinate system (the local coordinate system fixed to the turntable 112) can be calculated by the anterior resection method of FIG. 9B. Then, if the positions of many feature points are known, the camera position is calculated by the posterior resection method of FIG. 9A based on those positions.

Actually, the formula for calculating the exterior orientation parameters of the cameras 131 to 133 by the series of methods described above is a formula having many matrix elements including the coordinates of many feature points. In this calculation formula, the exterior orientation elements of the cameras 131 to 133 are obtained by repeating calculations until the unknowns converge. Also, bundle adjustment calculation is performed by the bundle adjustment calculator 297 at an appropriate timing to minimize the error.

The feature point extraction unit 250 extracts feature points from the images captured by the cameras 131-133. A feature point is a point that can be distinguished from the surroundings. For example, an edge portion, a portion having a different color or brightness from the surroundings, and the like are extracted as feature points. Extraction of feature points is performed by software processing. Differential filters such as Sobel, Laplacian, Prewitt, and Roberts filters are used to extract feature points.

The corresponding point identification unit 260 identifies correspondence relationships between feature points individually extracted from two or more images taken from different viewpoints. That is, the same feature points as the feature points extracted in one image are identified in the other image. The process of specifying the correspondence between feature points is performed using, for example, SIFT, SURF, Kaze, etc. as Feature-Based (feature extraction) methods, and template matching as Area-Based (Area Correlation) methods. will be Examples of template matching include a sequential similarity detection algorithm (SSDA), a cross-correlation coefficient method, and the like. The identification of corresponding points is basically performed for two images or two point cloud position data, but it can also be performed for three or more images or three or more point cloud position data.

As for the technology for obtaining the correspondence between a plurality of images, for example, the technology described in JP-A-2013-178656 and JP-A-2014-35702 can be used. In addition, Japanese Patent Application Laid-Open Nos. 2013-178656 and 2014-35702 also describe techniques for extracting feature points and techniques for obtaining three-dimensional positions of feature points. It can be used for the techniques described in the specification.

The three-dimensional coordinate calculation unit 270 calculates the three-dimensional positions of the feature points extracted from the image by the feature point extraction unit 150 by the anterior resection method. By calculating the three-dimensional positions of a large number of characteristic points that form the object, point cloud position data that enables the object to be grasped as a set of points whose three-dimensional coordinates are specified can be obtained. Furthermore, a three-dimensional model can be created based on this point cloud position data.

FIG. 9B shows the principle of anterior resection. In the anterior resection method, the direction toward the unknown point P is observed from a plurality of different points (two points _O1 and _O2 in the case of FIG. 9B), and the intersection of those direction lines is the unknown point P. find the position.

FIG. 9(B) shows the case where the overlapping area is photographed from different camera positions _O1 and _O2 using cameras with known exterior orientation parameters. Here, the screen coordinates of the unknown point P in one image are _p1 , and the screen coordinates of the unknown point P in the other image are _p2 . Since the camera positions _O1 and _O2 are known, and the camera postures at those positions are also known, the coordinates of the intersection point P of the direction line connecting _O1 and _p1 and the direction line connecting _O2 and _p2 are can be calculated.

The three-dimensional model creation unit 280 creates a three-dimensional model based on three-dimensional point cloud position data obtained by analyzing a large number of captured images. For example, three-dimensional point group position data consisting of three-dimensional coordinates of a large number of obtained feature points is used to create tin (irregular triangular network) to create a three-dimensional model of the object to be photographed. Techniques for creating a three-dimensional model based on point cloud position data are described, for example, in WO2011/070927, JP-A-2012-230594, and JP-A-2014-35702. As the three-dimensional coordinate system to be used, for example, local coordinates with the position of the rotation center of the turntable 112 as the origin are used.

The storage unit 290 stores various data used in the three-dimensional measuring apparatus 200, operation programs, various data obtained as a result of operations, and the like. It is also possible to use an external storage device (external hard disk device, data storage server, etc.) in addition to the storage unit 290 .

The rotation control unit 295 sends a control signal to a motor driving device 140 (see FIG. 7) that drives a motor (not shown) that rotates the turntable 112 to control the operation of the motor. The rotation control unit 295 functions to set the rotation timing and rotation speed of the turntable 112 . The rotation of the turntable 112 is detected by a rotary encoder, and the rotation control section 295 controls the rotation of the turntable 112 based on the detected value.

The shooting timing control unit 296 sends control signals for determining shooting timings to the cameras 131, 132, and 133, and controls shooting operations. The cameras 131 , 132 , 133 are controlled by the photographing timing control unit 296 to photograph the rotating turntable 112 and the plant 120 placed thereon. For example, the turntable 112 is rotated at a rotational speed of 5 to 10 rotations/minute, and each of the cameras 131, 132, and 133 repeatedly captures still images at intervals of 0.1 to 0.5 seconds.

4. Example of Processing Hereinafter, an example of processing for obtaining point group position data of the plant 120 on the turntable 112 and further creating a three-dimensional model of the plant 120 based on this point group position data will be described. The following processing is performed using the three-dimensional measurement system 100. FIG. A program for executing the processing described below is stored in the storage unit 290 and executed by the three-dimensional measuring device 200 . The program may be stored in an appropriate storage medium, storage server, or the like, and may be provided therefrom.

(photograph)
First, the positions and attitudes of the cameras 131 to 133 are set so that the turntable 112 and the random dot columns 121 to 123 are captured. Take pictures on it. Photographing is performed by the cameras 131 to 133 at specific time intervals while rotating the turntable 112 on which the plant 120 to be measured is placed at a constant speed.

The photographing is performed, for example, under the condition that 72 still images are photographed at equal time intervals while the turntable 112 rotates once at a constant speed, that is, photographing is performed 72 times from different angles of 5°. In this case, 72×3=216 images are taken from the three cameras 131-133. This image data is sent to the three-dimensional measuring device 200 and received by the image data receiving unit 210 . The image data accepted by the image data accepting portion 210 is stored in the storage portion 290 . The rotation speed and shutter interval can be arbitrarily set according to the type of plant and the point cloud density to be acquired.

After obtaining the image data of the photographed plant 120, the background is removed from each photographed image. This process is described, for example, in JP-A-2018-197685.

In this way, three sets (72.times.3=216 images) of photographed images are obtained from directions obtained by equally dividing the circumference of 360.degree. Processing is performed based on these 216 images.

(Identification of camera position)
The processing related to specifying the camera position is performed in each of the cameras 131-133. First, a plurality of camera positions at equiangular angular positions are calculated using the method shown in FIG. 5 (for example, FIG. 5 shows a case where 12 camera positions are selected).

Here, camera positions at 12 equiangular (30°) positions are specified using the method of FIG. FIG. 10 shows an example of the procedure of this processing. First, 12 camera positions shown in FIG. 5 are selected (step S101). For example, 12 camera positions with angular positions different by 30° when viewed from above are selected. The selection of the camera position at this stage is performed based on the rotational position of the turntable 112 and the photographing timing of the cameras 131-133. This process is performed by the camera position selection section 220 in FIG.

Next, a stereo image is selected from the images taken from the position selected in step S101 (step S102). In this case, considering the camera positions, the camera positions adjacent to (1) and (3), (3) and (5), (5) and (7), etc. are selected as the camera positions of the stereo image. be. That is, two images taken from camera positions (1) and (3) form a stereo image, two images taken from camera positions (3) and (5) form a stereo image, and a stereo image is formed from camera positions (5). ) and (7) constitute a stereo image, and a plurality of sets of stereo images are selected. This processing is performed by the stereo image selection unit 221 in FIG.

Next, feature points are extracted from the selected stereo image (step S103). This processing is performed by the feature point extraction unit 250 . The feature points extracted here include the feature points of the plant 120, turntable 112, and random dot columns 121-123. Next, the correspondence between feature points between stereo images is specified (stereo matching) (step S104). Also, the target 113 that appears in common between the stereo images is detected (step S105).

Next, relative orientation for calculating the relative positional relationship between the camera selected in step S101 and the feature points is performed by the backward resection method using the feature points for which the corresponding relationships are specified (S106). At this time, at the same time, the relative positional relationship between the corresponding points captured in the stereo image and the camera position is specified using the anterior resection method.

Next, the relative positional relationship obtained in step S106 is given the actual size from the scale obtained from the feature points and the position of the target 113, and absolute orientation is performed (step S107). Through this processing, the exterior orientation elements (position and orientation) of the camera and the positions of the feature points whose correspondence relationship was identified in step S104 are obtained. Finally, bundle adjustment calculation is performed to minimize the error (step S108).

The processing of S106 to S108 is performed by the exterior orientation element calculator 245, the three-dimensional coordinate calculator 270 and the bundle adjustment calculator 297 in FIG. The camera position is calculated by the above processing. For example, the positions of the cameras in (1), (3), (5), (7), . . . (23) in FIG. 5 are calculated. The above processing is performed in each of the cameras 131-133.

By repeating the above processing, the camera positions are calculated for all images while gradually increasing the camera positions.

For example, after performing the processing of S101 to S106 in FIG. 5, as shown in FIG. 6, the processing of S101 to S106 is repeated for 12 new camera positions. In this case, the newly obtained camera positions are (2), (4), (6), (8), (10), . . . (24) shown in FIG. (1), (3), (5), (7), . . . (23) shown in FIG. In this case, the camera positions (1), (3), (5), (7), . . . (23) shown in FIG. ), (8), (10), (24) are calculated. At this time, camera positions (1), (3), (5), (7), . . . (23) treated as initial values are recalculated.

In the above processing, the camera positions that have already been obtained are used as initial values, and all the camera positions that are the target at that time are calculated. This calculation is repeated with increasing camera positions until all images are covered. In this way, camera positions in all images are calculated.

For example, consider a case where an object to be measured placed on a turntable is photographed from 72 equiangular camera positions. In this case, the processing shown in FIGS. 5 and 6 is repeated six times to obtain the camera positions of all 72 images. Specifically, the processing to obtain 12 camera positions at intervals of 30° is shifted by 15° from the first time to the second time, shifted by -10° from the second time to the third time, and from the third time to the fourth time. Shift by 15° when transitioning to 1, shift by -10° when transitioning from 4th to 5th, and shift by 15° when transitioning from 5th to 6th. In this way, camera positions are obtained for all 12×6=72 images. It should be noted that the multiple camera positions with respect to the object to be measured do not necessarily have to be equiangular angular positions.

(Three-dimensional measurement of measurement object)
After obtaining the camera positions with respect to the turntable 112 in all the images, a stereo image is constructed from the adjacent camera positions, and the three-dimensional coordinate positions of feature points extracted from the stereo image are calculated. At this time, the rough position data of the point cloud obtained by the processing of FIG. 10 is also used. FIG. 11 is an example of a flowchart of this process.

When the process is started, two images obtained by photographing the plant 120 from adjacent or adjacent viewpoint positions (camera positions) from adjacent camera positions are selected as stereo images (step S201). A total of 216 images captured by the cameras 131 to 133 are used as candidates for selection. This processing is performed by the stereo image selection unit 221 . The two images to be selected may be captured by different cameras. It is also possible to select three or more images obtained by photographing overlapping objects as stereo images.

Next, feature points are extracted from the stereo image selected in step S201 (step S202). This processing is performed by the feature point extraction unit 250 in FIG. After the feature points are extracted from the stereo images, the correspondence between the feature points in the stereo images is specified (step S203). This processing is performed by the corresponding point identification unit 260 in FIG. Also, the target 113 is detected from the stereo image (step S204). This processing is performed by the target detection unit 240 . Target detection may be performed between steps S202 to S205.

Next, the three-dimensional position of the feature point for which the correspondence between the stereo images has been specified in step S203 is calculated (step S205). At this time, the feature points whose positions are specified in the process of FIG. 10 are used as initial values. This processing is performed by the three-dimensional coordinate calculation unit 270 . The three-dimensional position of the feature point is calculated as coordinates in a coordinate system fixed to the turntable 112 . The feature points whose positions are specified form a three-dimensional point group, and point group position data of the plant 120 is obtained.

Next, the process returns to the previous stage of step S201, and the processes after step S201 are repeated.

For example, suppose camera positions (1) and (2) are adjacent, and (2) and (3) are adjacent. In this case, it is assumed that images taken from camera positions (1) and (2) in FIG. ). In this case, camera positions (2) and (3) in FIG. 4 are selected in the processing from step S201 of the next flow (assuming that (2) and (3) are adjacent).

Thus, when a new combination of stereo images is selected, an unknown corresponding point (a corresponding unknown feature point that is common among the images forming the stereo image) is extracted at that stage, and its position is calculated. . By repeating this process, the three-dimensional positions of common feature points in stereo images sequentially selected from all images to be used are calculated. In this way, point group position data is obtained in which the plant 120, which is the object of three-dimensional measurement, is captured as a set of points whose three-dimensional coordinates are known.

After obtaining the point group position data of the plant 120, a three-dimensional model of the plant 120 is created based on the data. Since this process is a known process, a description thereof will be omitted. Creation of a three-dimensional model is described, for example, in International Publication No. WO2011/070927, Japanese Patent Application Laid-Open No. 2012-230594, Japanese Patent Application Laid-Open No. 2014-35702, and the like.

(Conclusion)
In this embodiment, the number of camera positions is gradually increased from 12 directions to 24 directions to . . . are obtained sequentially. At this time, the camera position to be selected is closed in a closed loop, and the calculation is repeated while gradually shifting it, and finally the camera position related to all images is obtained.

This method avoids the error accumulation problem. In addition, stereo images with a long base line such as the camera positions (1) and (2) in FIG. 5 are used as the initial values, and the whole is closed loop, so the accuracy is high. Also, since the result of the bundle adjustment calculation in each stage is used as the initial value in the next stage, convergence is not slowed down or broken.

In addition, when obtaining the camera position, corresponding points are also obtained by sparse stereo matching, and the error thereof is also minimized by bundle adjustment calculation. After obtaining the camera positions, stereo matching of adjacent camera positions is performed again to obtain a dense 3D point group. , which is the constraint point. For this reason, the calculation for obtaining a dense three-dimensional point group also converges quickly and the calculation accuracy is high.

(others)
It is also possible to control the rotation of the turntable 112 precisely, stop the rotation during photographing, and maintain a stationary state. For example, this method can be used when a long exposure time is required during shooting, or until the shaking caused by the vibration of the object converges. Also, the object of measurement is not limited to plants.

It is also possible that the positions of the cameras surrounding the turntable are not equiangular. In addition, in the selection of multiple camera positions surrounding the turntable, it is possible that the number of camera positions is not the same between the N-th time and the N+1-th time. In addition, in the selection of camera positions at a plurality of locations surrounding the turntable, it is also possible to have a configuration in which some of the camera positions overlap between the N-th time and the N+1-th time.

Alternatively, the object to be measured may be fixed without being rotated, and the camera directed toward the object to be measured may be moved on the circumference of the circle while shooting. In this case, the angular position of the camera is grasped with reference to the center of the circumference along which the camera moves. Also, the positional relationship between the measurement target and the camera may be relatively fixed. In this case, a large number of cameras are arranged on the circumference of the object to be measured, and images are taken from each camera position. In this case, the angular position is grasped with reference to the center of the circumference where the camera is arranged.

(Modification 1)
A mode is also possible in which the identification of camera positions shown in FIG. 10 is not performed for all camera positions. For example, at the stage where eight camera positions surrounding the object to be measured (rotary table) shown in FIG. 4 have been obtained, the process of sequentially obtaining adjacent camera positions shown in FIG. 2 may be performed. In this case, the unknown camera positions is calculated and newly extracted feature points are calculated.

(Modification 2)
The bundle adjustment calculation may not be performed at each stage, and the bundle adjustment calculation may be performed after all camera positions have been calculated.

(Modification 3)
In the case of FIG. 3, the pairs of camera positions (1) and (17), (9) and (25) may be selected as stereo images. In other words, as the stereo images selected from the four selected camera positions (1), (9), (17), and (25), images at non-adjacent camera positions may be selected. This is the same for FIGS. 4 to 6 as well.

(Modification 4)
The arrangement of the cameras as shown on the right side of FIG. 12 is effective for shooting without blind spots. In the method shown in FIG. 12, at least four cameras are arranged above and below, the lower two cameras face diagonally upward, and the upper two cameras face diagonally downward. Then, the upper side of the object to be measured is photographed by a diagonally downward pointing camera arranged relatively upward and an obliquely upward pointing camera arranged relatively upward. Then, the lower side of the object to be measured is photographed by a camera directed obliquely downward and a camera directed obliquely upward located relatively downward. Of course, the photographing ranges of both sets in the vertical direction partially overlap. By doing so, it is possible to suppress the occurrence of blind spots in the shooting range in the vertical direction.

100... 3D measurement system, 111... Base, 112... Rotating table, 113... Target, 120... Plant which is an object to be measured, 121 to 123... Random dot column, 124... Bowl, 125... Two-dimensional barcode display, 131- 133... Camera, 200... 3D measurement processing device using PC.

Claims (10)

an image data reception unit that receives a plurality of image data obtained by photographing an object to be measured and a random dot pattern from a plurality of surrounding viewpoints with a camera;
a camera position selection unit that selects three or more camera positions surrounding the measurement object from among the positions of the cameras that captured the plurality of images;
a stereo image selection unit that selects a stereo image from images captured from the camera position selected by the camera position selection unit;
a camera position calculation unit that calculates the position of the camera that captured the stereo image by the resection method using the stereo image,
The three-dimensional measuring apparatus, wherein the camera positions are selected a plurality of times, and at least some of the selected camera positions do not overlap at different times of selection. an image data reception unit that receives a plurality of image data obtained by photographing an object to be measured and a random dot pattern from a plurality of surrounding viewpoints with a camera;
a camera position selection unit that selects three or more camera positions surrounding the measurement object from among the positions of the cameras that captured the plurality of images;
a stereo image selection unit that selects a stereo image from images captured from the camera position selected by the camera position selection unit;
a camera position calculation unit that calculates the position of the camera that captured the stereo image by the resection method using the stereo image,
The three-dimensional measurement apparatus, wherein the selection of the camera positions is performed a plurality of times, and the selected camera positions do not overlap at different times of selection. In the camera position selection unit,
a first selection of selecting N camera positions at angular positions of 360°/N (N is a natural number of 3 or more) as viewed from the measurement object;
a second selection of selecting N camera positions from a position shifted by 180°/N from the N camera positions selected at the first time, as viewed from the object to be measured; 3. The three-dimensional measuring device according to claim 1 or 2. 4. The random dot pattern according to any one of claims 1 to 3, wherein the random dot pattern is formed on a surface of a longitudinal member fixed to a table on which the measurement object is placed and extending upward from the table. Three-dimensional measuring device. 5. The three-dimensional measuring apparatus according to claim 4, wherein the dimension in the height direction of said longitudinal member is greater than the dimension in the height direction of said object to be measured placed on said table. 6. The three-dimensional measuring apparatus according to claim 4, wherein said table is rotatable. A plurality of image data obtained by photographing the measurement object and the random dot pattern from a plurality of surrounding viewpoints by photographing the measurement object with the camera while rotating the base with respect to the camera. is obtained,
7. The three-dimensional measuring apparatus according to claim 6, wherein the center of rotation of the object to be measured is positioned within a polygon whose vertices are three or more positions surrounding the object to be measured. an image data receiving step of receiving a plurality of image data obtained by photographing an object to be measured and a random dot pattern from a plurality of surrounding viewpoints with a camera;
a camera position selection step of selecting three or more camera positions surrounding the measurement object from among the positions of the cameras that captured the plurality of images;
a stereo image selection step of selecting a stereo image from images captured from the camera position selected in the camera position selection step;
a camera position calculation step of calculating the position of the camera that captured the stereo image by the resection method using the stereo image;
A three-dimensional measurement method, wherein the camera positions are selected a plurality of times, and at least some of the selected camera positions do not overlap when different selections are made. A program that is read and executed by a computer,
an image data receiving step of receiving a plurality of image data obtained by photographing an object to be measured and a random dot pattern with a camera from a plurality of viewpoints around the computer;
a camera position selection step of selecting three or more camera positions surrounding the measurement object from among the positions of the cameras that captured the plurality of images;
a stereo image selection step of selecting a stereo image from images captured from the camera position selected in the camera position selection step;
a camera position calculation step of calculating the position of the camera that captured the stereo image by the resection method using the stereo image;
The three-dimensional measurement program, wherein the camera positions are selected a plurality of times, and at least some of the selected camera positions do not overlap in different selections. The camera position selection unit performs a first selection of three or more camera positions surrounding the measurement object and a second selection of three or more camera positions surrounding the measurement object,
The position calculation unit of the camera,
calculating the camera position selected for the first time;
3. The three-dimensional measuring apparatus according to claim 1, further comprising: calculating the second selected camera position using the camera position obtained in the calculation as an initial value.

Images