JP2019032218A - Location information recording method and device - Google Patents

Abstract

To enable a storage of location information on a check point in a moving check object.SOLUTION: A location information recording method according to one aspect of the present invention comprises: a first step of acquiring a wide area image; a second step of acquiring a definition image with higher resolution than the wide area image; a third step of extracting a definition image characteristic point from the definition image; a fourth step of generating a three-dimensional point group giving a coordinate on a three-dimension to the definition image characteristic point from the definition image; a fifth step of extracting a wide area characteristic point from the wide area image, and matching the extracted wide area characteristic point with the definition image characteristic point; and a sixth step of recording the wide area image characteristic point in association with a three-dimensional coordinate of the definition image characteristic point in which the matching to the wide area image characteristic point is established.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a technique for storing position information in space.

  In recent years, inspection using a UAV (Unmanned Aerial Vehicle) has become common. The reason for the generalization is that the inspection using the UAV can be carried out in a shorter time and at a lower price than the inspection by hand as the price of the UAV body and sensor used for the inspection decreases. After the inspection, post-inspection work based on the inspection result is necessary. As the work after inspection, for example, repair of a damaged part, comparison of the same part, and re-inspection of the same part can be considered.

  In order to carry out this work, it is necessary to grasp the relative position information of the inspection location. There is a technique called SfM (Structure from Motion) as a technique for grasping the relative position of an imaging target using a camera and the camera itself. SfM is a technique for estimating the relative position relationship between the relative position of the camera and the three-dimensional shape of the object to be photographed from an image group obtained by photographing the common area. Therefore, when there is no image pair that captures a common area between image groups, the relative positional relationship of the cameras estimated in each image group is not known, and the relative 3D shape estimated between the image groups is not known. I don't know the position and relative scale.

  On the other hand, if a portion unnecessary for the inspection is photographed in detail like the inspection object, the calculation time becomes enormous as the number of images to be processed increases. On the other hand, in Japanese Unexamined Patent Application Publication No. 2014-06148 (Patent Document 1), a flying object equipped with an imaging device flies at a high altitude, acquires a wide area image including a measurement range, and obtains a three-dimensional image at the time of acquiring the wide area image. A step of acquiring a position, a step of setting a measurement range based on the wide area image, a step of acquiring a fine image at a predetermined distance interval when the flying object flies at a low altitude, and acquiring a position at the time of acquiring the fine image. Perform wide-area image / fine-image matching between the wide-area image and the fine image that have been scaled based on the process, the step of matching the magnification of the wide-area image and the fine image, the position at the time of wide-area image acquisition, and the position at the time of fine-image acquisition And a step of performing fine image / fine image matching of adjacent fine images based on relative information obtained by wide area image / fine image matching and creating a measurement range fine 3D model of the entire measurement range. It has been.

  FIG. 1 is a diagram showing the concept of Patent Document 1. In FIG. By grasping the relative scale using the wide area image, the relative position and relative scale of two or more three-dimensional shapes can be determined.

JP 2014-06148 A

  In Patent Document 1, absolute position information measured by a GPS (Global Positioning System) is used to collate a fine image and a wide area image. However, if the shooting target moves, the relationship between the absolute position of the shooting target when shooting with a wide-area image and the absolute position of the shooting target when shooting with a fine image changes, and therefore, verification using GPS is not possible. .

  For example, an offshore wind power generator or the like exists as a moving inspection target. Therefore, an object of the present invention is to provide a technique for automatically storing position information of an inspection point in a moving inspection object using a fine image and a wide area image.

  One aspect of the present invention is a first step of acquiring a wide area image, a second step of acquiring a fine image having a higher resolution than the wide area image, a third step of extracting fine image feature points from the fine image, A fourth step of generating a three-dimensional point group in which a three-dimensional coordinate is given to a fine image feature point from a fine image, extracting a wide area image feature point from the wide area image, and performing matching with the fine image feature point A position information recording method comprising: a fifth step, a sixth step of associating and recording the three-dimensional coordinates of the fine image feature points for which matching has been established with the wide-area image feature points.

  Another aspect of the present invention includes a wide-area image acquisition unit that acquires a wide-area image, a fine-image acquisition section that acquires a fine image with higher resolution than the wide-area image, and extracts fine-image feature points from the fine-image, A fine image processing unit that generates a three-dimensional point cloud that gives coordinates in three dimensions to a fine image feature point from an image, and extracts a wide area image feature point from a wide area image, performs matching with the fine image feature point, The position information recording apparatus includes a position information recording unit that records a three-dimensional coordinate of a fine image feature point for which matching is established in association with a wide area image feature point.

  The absolute positions of the shooting targets of the wide area image and the fine image are not necessarily equal, and even in an environment where the three-dimensional shape of the wide area image and the fine image cannot be aligned on a GPS basis, the inspection location can be grasped on the wide area image.

The conceptual diagram of the grasping | ascertainment of the relative scale using a wide area image. The system flow figure of Embodiment 1. FIG. The conceptual diagram of feature point matching. Fig. 3 is a conceptual diagram of generating a three-dimensional point cloud. The conceptual diagram of the feature point matching using area | region limitation and RANSAC. The conceptual diagram of a projection error and its correction. FIG. 2 is a functional block diagram of the first embodiment. The table of fine image DB. The functional block diagram of the feature point matching part between fine images. The table of image feature point DB. Table of matching DB. The functional block diagram of a SfM part. Table of camera DB. The table of 3D point cloud DB. The functional block diagram of the feature point matching part of a wide area image and a fine image. Table of wide area image DB. The table of sensor DB. The table of wide area image coordinate position DB. FIG. 6 is a functional block diagram of the second embodiment.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. It should be noted that this embodiment is merely an example for realizing the present invention, and does not limit the technical scope of the present invention. In each figure, the same reference numerals are given to common configurations.

  Notations such as “first”, “second”, and “third” in this specification and the like are attached to identify the constituent elements, and do not necessarily limit the number, order, or contents thereof. is not. In addition, a number for identifying a component is used for each context, and a number used in one context does not necessarily indicate the same configuration in another context. Further, it does not preclude that a component identified by a certain number also functions as a component identified by another number.

  The position, size, shape, range, and the like of each component illustrated in the drawings and the like may not represent the actual position, size, shape, range, or the like in order to facilitate understanding of the invention. For this reason, the present invention is not necessarily limited to the position, size, shape, range, and the like disclosed in the drawings and the like.

  In the present embodiment, a description will be given using inspection using UAV, which is an example of utilizing a position information recording apparatus. The position information recording apparatus according to the present embodiment records position information on a wide area image of an inspection location using a wide area image showing the entire inspection target and a fine image acquired from the camera when the inspection is performed. . However, the UAV is an example, and can be attached to a manned airplane, a robot moving on the ground, or a person. The sensor used for the inspection may use the photographing result of the camera constituting the apparatus as it is. It is also conceivable to prepare a sensor different from the camera and give position information to the inspection result acquired by the sensor.

  In the present embodiment, it is also assumed that the object to be imaged is not stationary and moves between the past inspection timing and the latest inspection timing. Further, at the timing of each inspection, it is assumed that a fine image corresponding to a partial area of the wide area image is acquired by, for example, UAV. The wide area image is a remote image obtained by photographing a photographing object from a high altitude by, for example, UAV, and the fine image is a close image photographed closer to the photographing object. It is assumed that the fine image includes a plurality of still images taken from a plurality of camera positions at a certain inspection timing, and the subject to be photographed does not substantially move between the plurality of still images. Such a still image can be extracted from, for example, a moving image captured while the UAM is flying close to the object to be imaged. On the other hand, the wide area image may be taken and stored only once before the first inspection timing, or may be taken together with the fine image at each inspection timing.

  In the embodiment described in detail below, a three-dimensional map of an object to be imaged is created using a technique such as Visual SLAM (Simultaneous Localization and Mapping) for fine images. To do. As is well known, in Visual SLAM, by analyzing a moving image, it is possible to sequentially estimate the position / posture of the camera and the three-dimensional map of the photographing target. In this embodiment, the estimated 3D map is compared with the wide area image, and the feature points are associated with each other, thereby estimating and storing the position of the 3D map in the coordinate system of the wide area image.

  By this method, the three-dimensional coordinates obtained from the fine image can be associated with the coordinate system of the wide area image. Therefore, even if the absolute position of the subject to be captured included in the fine image changes between the fine image at the past inspection timing and the fine image at the current inspection timing, the subject to be photographed in the same coordinate system. It becomes possible to specify the position of.

  A first embodiment of the present invention will be described with reference to FIG. FIG. 2 is a processing flowchart of the position information storage system according to the first embodiment of the present invention. In the present embodiment, feature point matching is performed between a plurality of temporally continuous fine images in order to grasp the relative positional relationship of the cameras that have acquired the fine images on the wide area image (S201). The plurality of fine images are taken at short time intervals, and the subject to be photographed is not substantially moved within that time.

  The concept of feature point matching will be described with reference to FIG. Matching is an operation for connecting the same points appearing in two images. First, the feature point 301 of the fine image 300 acquired by the camera is acquired (FIG. 3A).

  The feature point 301 is a point on the image that can be easily distinguished from other points, and the gradient information of the pixel information can be expressed as the feature amount of the feature point. Examples of feature quantities include known SIFT, SURF, ORB, AKAZE, and the like. However, when the position of the camera does not change and the image hardly changes, it is not necessary to acquire feature points from the image. A lot of research has been done on determining whether to acquire a feature point, so the details are omitted.

  Next, the acquired feature points are matched. The feature point matching is to link points representing the same point from the two fine images 300a and 300b (FIG. 3B). When the feature quantities of the compared feature points indicate similar values, it is determined that the two feature points 301a and 301b represent the same point. In order to increase the accuracy of the matching, an incorrect matching may be eliminated by using a geometric constraint or the like. A means for deleting an incorrect matching will be described later.

  Next, using the result of feature point matching, the relative position of the three-dimensional point group generated from the initial fine image, the relative position of the camera, and the direction of the camera are estimated (S202). The three-dimensional point group is obtained by giving a three-dimensional coordinate to a point on the image. The camera relative position and orientation are the position and orientation of the camera when the image is taken.

  As shown in FIG. 4, the relative position of the initial three-dimensional point group 401, the relative position 402 of the camera, and the direction 403 can be estimated by using triangulation based on the result of feature point matching. Such processing can be performed by the aforementioned Visual SLAM.

  However, the process using triangulation is performed on the assumption that the camera parameters are known. The camera parameter is a parameter expressing the number of pixels of the image, the sensor size, the focal length, the lens distortion, and the like. However, in practice, camera parameters may be unknown in advance, or may include errors even if estimated. Even in a situation where the camera parameters cannot be accurately grasped, the camera parameters can be estimated from the matching result of the feature point matching. Therefore, it is not always necessary to estimate the camera parameters in advance.

  Next, feature point matching between the wide area image and the fine image is performed (S203). First, feature point matching between a captured wide area image and a fine image with different altitudes will be described. In the feature point matching, as described in the matching between the fine images, it is determined whether the two points indicate the same location based on the similarity of the feature amount based on the pixel gradient information. The matching based on the similarity includes many miscorresponding points. Therefore, a known algorithm called RANSAC (RANdom Sample Consensus) is used in order to estimate the constraint equation behind and implement correct feature point matching accuracy. Thereby, the matching which matches the same point appropriately is extracted.

  In RANSAC, a constraint that is stochastically behind is estimated from data including an outlier, and whether matching is correct is determined based on the constraint. However, in images with different altitudes, the constraint conditions existing behind cannot be estimated due to the high proportion of false correspondences, and feature point matching cannot be performed well.

  A feature point matching method between a fine image and a wide area image will be described with reference to FIG. As an example, FIG. 5 attempts to acquire correct matching by using the constraint that the size of the fine image is changed and the rotated image is associated with the wide area image.

  As shown in FIG. 5A, the wide area image 500 is photographed at a position higher than the portion where the fine image 300 was photographed. As a result, the photographing object appears as if the fine image 300 is made smaller and rotated. However, as a result of a large amount of incorrect matching, it is difficult to estimate how much the size has been changed and rotated by looking at the matching result alone.

  FIG. 5B shows an example in which feature point matching is performed by limiting the region having the feature point 5001 to be matched to 510 in the wide area image 500. In this way, if an obvious matching error is removed based on the location of the fine image 300 in the wide area image 500, it is easy to estimate changes in size and rotation. Then, as a result of estimating the back restraint, it is possible to remove the false matching as shown in FIG.

  In the present embodiment, before executing RANSAC, the matching including the feature point 5001x outside the imaging range in the wide area image 500 is removed. By removing a large amount of false matching, we were able to correctly estimate the constraints that existed behind, and as a result, obtain the correct matching results. A specific procedure will be described below.

  First, a wide-area image 500 captured in advance is acquired from a wide-area image database (abbreviated as DB; the same applies hereinafter). However, the wide area image 500 may be taken before the fine image 300 is taken, and the fine image 300 may be taken as it is. Next, the feature point 5001 of the wide area image 500 is acquired and stored in the feature point DB. However, when the wide area image 500 is captured in advance, the feature point 5001 of the wide area image 500 may be acquired in advance and stored in the feature point DB.

  The area 510 of the wide area image 500 to be matched is previously stored in the wide area image DB as an initial three-dimensional point cloud area, and the matching area is limited based on the area. The coordinates of the initial three-dimensional point area are, for example, the coordinates of the shooting start position of the fine image. As the coordinates, camera coordinates recorded by GPS or the like at the time of capturing a fine image may be used, or camera direction information may be used in combination. According to these pieces of information, it is possible to roughly specify the place where the camera has taken a picture. In this case, precise coordinates are not necessary. If the camera coordinates are not recorded, the camera position coordinates estimated in step S202 may be used.

  The range of the region is defined as a circle having a predetermined radius and a square having a predetermined side length with the coordinates as the center. As another method, in order to confirm the initial three-dimensional point cloud region from the wide area image 500, a mark indicating the initial three-dimensional point cloud region is placed in a region where the movement of the photographing target is assumed in advance, and the mark is photographed. Thus, it can be considered that the limited area is not stored in advance.

  As another method, a wide area image may be displayed on the operator before matching, and the operator may specify the range. By limiting these areas, erroneous matching is removed.

  Next, matching using RANSAC is performed. A possible constraint is, for example, a plane constraint that is a constraint condition on an image at the same point in two images obtained by photographing points existing on the same plane. Epipolar constraints may also be used. Only matching that satisfies the plane constraint estimated from the matching result of the wide area image 500 and the fine image 300 is processed as correct matching.

  When there is a feature point 301 of the fine image that matches the feature point 5001 of the wide area image, the three-dimensional point group 401 of the feature point 301 can confirm the correspondence with the point on the wide area image. However, in the estimation of the three-dimensional point group using only the fine image, the three-dimensional point group 401 may be distorted. Therefore, the feature points of the wide area image and the fine image are recorded by correcting the three-dimensional point group and the camera position (processing S204). A method for correcting the three-dimensional point group and the camera position will be described below.

  With reference to FIG. 6, the distortion of the three-dimensional point group and this correction will be described. Among the matching feature points of the wide area image 500 and the fine image 300, those whose three-dimensional coordinates are calculated by triangulation are extracted. Based on the correspondence of the feature points in the wide area image 500, the camera positions of the three-dimensional point group and the fine image are corrected.

  First, the place where the wide area image is photographed is estimated from the position information of the three-dimensional point group 401 specified on the wide area image. From the estimated result, the relative self-position is corrected by adjusting the positions of the camera position 402 and the three-dimensional point group 401 as indicated by the thick arrows in the figure so that the sum of the projection errors of the wide area image and the fine image is minimized. A point indicated by 301 is a place where a feature point exists on the image, and points indicated by 402 and 401 are relative positions of the camera and the three-dimensional point group.

  The location where the estimated wide-area image is captured includes an error, and the coordinates on the image estimated from the camera position and the relative position of the three-dimensional point group do not match the coordinates on the actual image. The operation of estimating where the straight line connecting the camera position and the three-dimensional point group exists on the fine image 300 is called reprojection. The reprojected location can be calculated from the camera position, the camera orientation, the camera parameters, and the relative position of the three-dimensional point group. An error between the re-projected point 6002 and the point 6001 reflected in the actual image is referred to as a projection error 6003. Then, the estimation error is reduced by correcting the camera position and the three-dimensional point group position so as to minimize the projection error. As a means for minimizing, for example, bundle adjustment which is a nonlinear optimization method is used.

  The feature points of the initial three-dimensional point group obtained based on the initial fine image are associated with the feature points of the wide area image. If coordinate axes are defined on the wide area image, the feature points of the initial three-dimensional point group can be represented by the coordinates on the wide area image. In the present embodiment, the fine image is, for example, a plurality of still images taken by the UAV while flying around the photographing object at a certain inspection timing, and the initial fine image is, for example, 2 selected from among them. There are two still images. When processing is performed in real time at the time of shooting, the two still images are two fine images immediately after the start of shooting. In the following description, real-time processing will be described as an example. However, it is also possible to save an image once and perform batch processing later.

  Next, the UAV moves from the starting point. The UAV takes a new fine image while moving. Feature point matching is performed between the added new fine image and the previous fine image (step S205). When there is a point to which position information is added as a three-dimensional point group among the feature points that have been feature point matched, the camera position of the additional fine image is estimated based on the position information of the three-dimensional point group ( Process S206). However, after the camera position is estimated, a correction process using a projection error may be performed. In order to share a feature point with the same position information added between two fine images, the two fine images may be taken so that the photographing ranges partially overlap.

  Subsequently, a three-dimensional point group is added (processing S207). Of the feature points extracted as a result of the feature point matching (process S205), feature points to which relative position information is not added are extracted. For feature points to which the position information is not added, relative position information is added by using triangulation as in step S202. Even after the process is completed, a process of correcting using the projection error may be performed as in the process S204. The feature point to which the relative position information is added is added to the three-dimensional point group.

  Subsequently, feature point matching between the wide area image and the new fine image is performed (steps S208 to S209). At the start point, in order to make feature point matching successful, a matching area is determined by using known information or a mark (processing S203). The matching area between the new fine image and the wide area image may be determined in the same manner as in the process S203, but may be determined by the camera position and camera parameters of the new fine image (process S208). That is, since the camera position of the fine image is estimated in step S206, the fine image position in the wide area image can be estimated based on the camera parameters such as the magnification.

  After performing the feature point matching, if necessary, a correction process using a projection error is performed in the same manner as in step S204, and the 3D point group and the camera position are corrected to record the feature points of the wide area image and the fine image. (Processing S210).

  Thereafter, even if the absolute position information is not known by repeating the processes of step S205 to step S210, the absolute position information by GPS or the like cannot be acquired by performing feature point matching in which the regions are sequentially limited. Even under circumstances, the correspondence between the position information of the wide area image and the fine image can be continued.

  A method for recording the inspection result data together with the position information of the inspection object will be described. In the process from step S205 to step S210, only the relative position information of the camera position and the three-dimensional point group of the feature points photographed by the camera is recorded. In the inspection automation process which is the object of the present embodiment, it is necessary to record the relative position information of the inspection object and sensor data reflecting the state of the inspection object in association with each other.

  In one example, the relative position information of the inspection object to be recorded can be determined depending on the sensor used for the inspection. When using a sensor in which the position of the sensor coincides with the position of the measurement object, such as a sensor that detects temperature or a specific substance, the sensor position may be recorded as position information. When the sensor acquires information on a position away from the sensor, such as a camera, a spectrum camera, or a thermal infrared camera, and the sensor data is expressed as a two-dimensional image, the sensor (camera) The position information to be recorded differs depending on the orientation of the. When the sensor is oriented in the same direction as the camera for recording position information, the relative position of the three-dimensional point group acquired by the camera may be recorded. When the directions are different, it is necessary to grasp where the data acquired by the sensor corresponds on the three-dimensional coordinates. For this purpose, the position can be determined by grasping the position of the inspection object from which the sensor data is acquired based on the position of the sensor itself and the sensor parameters such as the direction and sensitivity of the sensor. In addition, the distance from the inspection object can be grasped by providing the sensor with a distance sensor. It can also be limited by assuming an existing plane.

  With reference to FIG. 7, functional blocks for implementing the procedure of FIG. 2 will be described. The position information recording apparatus 700 of this embodiment includes a camera 14001, a fine image DB 14002, a feature point DB 403, a matching DB 404, a three-dimensional point group DB 405, a camera DB 406, a wide area image DB 407, and a wide area image coordinate position DB 408. A sensor DB 409, a fine image processing unit 410, a position information recording unit 413, and a result output unit 417.

  Such a functional block can be basically composed of a general server including an input device, an output device, a storage device, and a processing device. In the present embodiment, functions such as calculation and control are realized in cooperation with other hardware by executing a program stored in the storage device by the processing device. A program executed by a server, its function, or means for realizing the function are called "part", "function", "means", "unit", "module", etc., and may be described as the subject of processing. . In the following description, data stored in the storage device may be described in terms of “˜table”, “˜list”, “˜DB”, “˜queue”, “information”, etc. As long as is equivalent information, the data format is not limited.

  In addition, the above configuration may be configured by a single server, or any part of the input device, output device, processing device, and storage device may be configured by another computer connected via a network. Good. In the configuration of FIG. 7, the camera 14001 is included in the apparatus for the sake of explanation. However, the camera 14001 is separately mounted on the UAV and configured to transmit image data to the position information recording apparatus 700 from a remote location via a wireless line or the like. be able to.

  The position information recording apparatus 700 acquires a fine image or a moving image for acquiring position information with one or a plurality of cameras 14001, and stores the captured image in the fine image DB 14002.

  FIG. 8 shows an example of the data structure of the index of the fine image DB 14002. For the fine image, for example, the fine image ID 801 is attached in the order of photographing so that the order of acquisition is known. Alternatively, the shooting time may be stored. If a camera is used, the camera ID 804 of the photographed camera is stored. Further, when shooting conditions such as the camera position 802 and the camera orientation 803 at the time of shooting are known, these may be saved. The camera position can be acquired by GPS, for example. However, even when these are not known, the camera position 802 and the camera orientation 803 can be estimated and stored as described above. Corresponding to the camera ID 804, camera performance parameters such as the number of pixels and focal length may be stored separately.

  FIG. 9 is a functional block diagram of the feature point matching unit 411 between fine images. In this embodiment, each functional block is realized by software. The saved fine images are subjected to feature point matching between the fine images by the feature point matching unit 411 between the fine images. The feature point matching unit between fine images includes an image feature point detection unit 501, an image feature point feature quantity comparison unit 502, an image feature point feature quantity calculation unit 503, and a feature point matching RANSAC unit 504.

  FIG. 10 shows an example of the data structure of the feature point DB 403. The image feature point detection unit 501 acquires a fine image from the fine image DB 14002, detects a feature point on the fine image, assigns a fine image ID 1001 and a feature point ID 1004, and stores the feature point in the feature point DB 403 together with the two-dimensional coordinates 1002 of the feature point. save. An image feature point feature amount calculation unit 503 calculates a feature point feature amount 1003 using the fine image and the two-dimensional coordinates 1002 of the feature points, and stores them in the feature point DB 403.

  The image feature point feature amount comparison unit 502 stores a combination of feature points having the feature amount 1003 in the feature point DB 403 close to the matching DB 404 as feature point matching.

  FIG. 11 shows an example of the data structure of the matching DB 404. In the matching DB 404, a fine image ID 1102 (1101) of a plurality of fine images and a feature point ID 1103 (1004) of feature points matched in the fine image of the fine image ID are stored in correspondence with the matching ID 1101. . The example of the first line in FIG. 11 indicates that the point with the feature point ID “6” of the image ID “2” matches the point with the feature point ID “39” of the image ID “3”.

  The feature point matching RANSAC unit 504 estimates a constraint condition behind the two images and removes erroneous matching from the matching DB 404.

  FIG. 12 shows the SfM unit 412. The SfM unit 412 estimates a three-dimensional point group, a camera relative position, a camera orientation, and the like.

  The initial three-dimensional point group generation unit 601 obtains the two-dimensional coordinates of the feature points to be matched based on the feature point DB 403 and the matching DB 404 and, as described with reference to FIG. Point cloud coordinates can be obtained. The initial three-dimensional point group coordinates are stored in the three-dimensional point group DB 405. The camera parameters of the camera DB 406 and the camera position of the fine image DB 14002 can be estimated. The camera parameters are stored in the camera DB 406, and the camera position and the like are stored in the fine image DB 14002 in the camera position 802 and the camera direction 803, respectively. Store like so.

  FIG. 13 shows an example of the data structure of the camera DB 406. For the camera ID 1301, camera parameters such as the number of pixels 1302, focal length 1301, sensor size 1304, and distortion coefficient 1305 are stored. These may be stored in advance if they can be stored in advance, or may be values estimated from fine images.

  FIG. 14 is an example showing the data structure of the three-dimensional point group DB 405. Each point of the three-dimensional point group generated by the initial three-dimensional point group generation unit 601 is assigned a three-dimensional point group ID 1401 that is a unique ID, and the three-dimensional point group ID 1401 is a source of the three-dimensional point group. It is stored together with a fine image ID 1402 of the fine image and a feature point ID 1403 of the feature points constituting the three-dimensional point group in the fine image. In addition, the relative position 1404 of the three-dimensional point group is stored. The relative position 1404 of the three-dimensional point group indicates relative position coordinates with respect to the origin (0, 0, 0) arbitrarily set for the three-dimensional point group, for example. In the example of FIG. 14, the data in the first row includes the feature point ID “33” of the fine image ID “1”, the feature point ID “22” of the fine image ID “3”, and the feature of the fine image ID “4”. The point ID “14” is matched, and the relative position coordinate of the point on the three-dimensional point group is (2, 3, 4).

  When it is desired to store the color information of the point group in the three-dimensional point group ID 1401, the color information is acquired from the fine image DB 402. However, when three or more detailed images are used, the camera parameters can be estimated even when there are no camera parameters prepared in advance in the camera DB 406. In that case, the estimated camera parameters are stored in the camera DB 406.

  The camera position estimation unit 603 uses the two-dimensional coordinates of the feature points in the feature point DB 403 based on the matching information in the matching DB 404, the relative position information in the three-dimensional point cloud database 405, and the camera parameters in the camera DB 406 to generate a new fine image. The camera position is estimated and stored in the fine image DB 402.

  The new three-dimensional point cloud adding unit 602 uses the two-dimensional coordinates of the feature points in the feature point DB 403 and the camera position of the new fine image in the fine image DB 402 based on the camera parameters of the camera DB 406 to obtain the three-dimensional point cloud DB 405. The relative three-dimensional position of the feature points not stored in is estimated and stored (added) in the three-dimensional point group DB 405.

  The bundle adjustment unit 604 calculates a projection error using the two-dimensional coordinates of the feature point DB 403, the three-dimensional point group ID, and the camera parameters of the camera DB 406, and corrects the three-dimensional point group, the camera position, and the camera orientation. These are stored in the three-dimensional point group DB 405 and the fine image DB 402, respectively.

  FIG. 15 shows a configuration block diagram of the feature point matching unit 414 for the wide area image and the fine image. The feature point matching unit 414 for the wide area image and the approaching image includes an image feature point detection unit 501, an image feature point feature amount comparison unit 502, an image feature point feature amount calculation unit 503, a feature point matching RANSAC unit 504, and feature points. A matching area limiting unit 701 is included. Except for the feature point matching area limiting unit 701, the configuration is basically the same as that of the feature point matching unit 411 for fine images shown in FIG. 9 except that the target image includes a wide area image. The feature point matching area limiting unit 701 determines a limited area for matching based on the position of the feature point included in the immediately preceding fine image on the wide area image. That is, since it is estimated that the feature point of the current fine image is located in the vicinity of the position of the previous fine image, the limited region may be determined based on the position of the feature point of the previous fine image. However, for the correspondence with the initial three-dimensional point group, the start coordinates described in the wide area image DB 407 are used.

  The wide area image coordinate position calculation unit 415 calculates a wide area image coordinate position indicating where the feature point captured in the fine image is associated with the coordinates on the wide area image. That is, the coordinates on the wide area image are associated with the feature points matched by the feature point matching unit 414 of the wide area image and the fine image. Note that the feature points extracted from the wide area image are recorded in the same manner as the feature point DB 403 in FIG.

  FIG. 16 shows an example of the data structure of the wide area image DB 407. For the wide area image ID 1601 of the wide area image, the camera position 1602 and the camera direction 1603 that captured the image are stored. The camera position 1602 can be recorded by acquiring the camera position by GPS or the like. The direction of the temple 1603 can be obtained from the operation parameters of the camera. Alternatively, these may be values estimated from the wide area image itself.

  The start position 1604 indicates the position of the region 510 in the matching between the wide area image and the fine image described with reference to FIG. As described above, the start position can be set and input artificially by the operator looking at the wide area image. Alternatively, a mark that can be confirmed on a wide area image is placed in the field, and the mark is extracted by image processing from the wide area image obtained by photographing the mark, and can be determined based on the mark. Alternatively, when the subject does not move, it can be set at a position estimated as the same location as the subject by GPS or the like. Or it can determine from the position and direction of the camera which image | photographed the fine image for extracting an initial three-dimensional point group.

  As a method for setting the start position 1604, two-dimensional coordinates indicating the center or corner of the area as shown in FIG. 16 may be used, or if the area is a rectangle, two-dimensional coordinates indicating the four corners may be used.

  FIG. 17 is an example showing the data structure of the sensor DB 409. A sensor ID 1701 that uniquely identifies the sensor, a definition 1702 of an inspection position acquired by the sensor, and a sensor orientation 1703 are included.

  FIG. 18 is an example showing the data structure of the wide area image coordinate position DB 408. A point having the three-dimensional point group ID 1401 shown in FIG. 14 corresponds to which of the feature points having the wide area image feature point ID 1803 of the wide area image having the wide area image ID 1802. Further, the wide area image coordinates 1804 on the wide area image at that point are associated.

  Finally, the result output unit can indicate where the inspection location is on the wide area image using the wide area image coordinates 1804 of the wide area image coordinate DB 408. In addition, it is possible to create a three-dimensional point cloud model in which the correspondence between the wide area image and each feature point is found. For example, there can be considered a method in which numbers are added to the points and feature points of the wide area image, and the points where the numbers match represent the same point.

  In the first embodiment, the process ends when the positional correspondence between the wide area image and the fine image is acquired. In actual inspection work, periodic inspections are performed and the latest results are compared with past results. By applying the first embodiment, the position of the inspection location in the fine image captured at different timings can be associated using the coordinates of the wide area image. Therefore, even when the absolute position of the inspection location is moved, the current and past inspection locations can be aligned, and the inspection contents that are different in time can be compared.

  FIG. 19 is a block diagram of the second embodiment. An inspection result comparison unit 801 is provided instead of the result output unit 417. Although the display of the position was the main purpose in the first embodiment, the main purpose in the second embodiment is to compare the inspection results using the recorded positions.

  In the first embodiment, the feature point matching unit 414 of the wide area image and the fine image processes the area limitation for the feature point matching of the wide area image and the fine image using only the image information. When acquiring fine images by performing continuous aerial photography using a UAV or the like, the positions of the fine images continuously shot are close to each other, so the position of the latest fine image in the wide-area image is estimated. If possible, an area for the next matching can be estimated. However, when the UAV or camera orientation changes suddenly, it may be difficult to limit the area.

  Therefore, by installing an acceleration sensor or magnetic sensor in the UAV or camera and using the acceleration information or magnetic information, the direction in which the UAV or camera suddenly changes direction is predicted and used for matching Can be corrected.

14001 Camera 14002 Fine image DB
403 Feature point DB
404 Matching DB
405 3D point cloud DB
406 Camera DB
407 Wide Area Image DB
408 Wide-area image coordinate position DB
409 Sensor DB
410 Fine image processing unit 411 Feature point matching unit 412 between fine images SfM unit 413 Position information recording unit 414 Wide area image and fine image feature point matching unit 415 Wide area image coordinate position calculation unit 416 Wide area image coordinate position recording unit 417 Result output Unit 601 initial three-dimensional point group generation unit 602 new three-dimensional point group addition unit 603 camera position estimation unit 604 bundle adjustment unit 701 feature point matching region limitation unit 801 inspection result comparison unit

Claims (10)

A first step of acquiring a wide area image;
A second step of obtaining a fine image having a higher resolution than the wide-area image;
A third step of extracting fine image feature points from the fine image;
A fourth step of generating, from the fine image, a three-dimensional point group in which three-dimensional coordinates are given to the fine image feature points;
A fifth step of extracting a wide area image feature point from the wide area image and performing matching with the fine image feature point;
A sixth step of recording the three-dimensional coordinates of the fine image feature points for which matching has been established in association with the wide-area image feature points;
A position information recording method comprising: In the fourth step, Visual SLAM is used.
The position information recording method according to claim 1. In the fifth step, an area for performing the matching is limited to a part of the wide area image;
The position information recording method according to claim 1. In the fifth step, in order to limit the matching area to a part of the wide area image, a mark included in the wide area image is used.
The position information recording method according to claim 3. In the fifth step, in order to limit the matching area to a part of the wide area image, a range can be specified by an operator.
The position information recording method according to claim 3. In the first step, a wide area image is acquired by aerial photography;
In the second step, a plurality of fine images are obtained by a series of aerial photographs,
In the fifth step, in order to limit the matching area to a part of the wide area image, the position information of the camera for performing the series of aerial photography is used.
The position information recording method according to claim 3. In the fifth step, the camera includes at least one of an acceleration sensor and a magnetic sensor, and uses at least one of acceleration information and magnetic information to estimate a region captured by the camera, Limiting the matching area to a part of the wide area image;
The position information recording method according to claim 6. In the sixth step, the coordinates on the wide area image of the wide area image feature points are recorded in association with the three-dimensional coordinates of the fine image feature points that have been matched.
The position information recording method according to claim 1. In the second step, fine images are acquired at different time timings for the same object, and the absolute position of the object is changing at the different time timings,
Detecting a temporal change of the object based on the information recorded in the sixth step;
The position information recording method according to claim 8. A wide area image acquisition unit for acquiring a wide area image;
A fine image acquisition unit for acquiring a fine image having a higher resolution than the wide-area image;
A fine image processing unit that extracts a fine image feature point from the fine image and generates a three-dimensional point group that gives a three-dimensional coordinate to the fine image feature point from the fine image;
A wide-area image feature point is extracted from the wide-area image, matched with the fine-image feature point, and the three-dimensional coordinates of the fine-image feature point that has been matched are recorded in association with the wide-area image feature point. Location information recording section,
A position information recording apparatus comprising:

Images