JP7467722B2 - Feature Management System - Google Patents

Description

This invention relates to a feature management system that uses point cloud data obtained by measuring features.

Laser surveying devices are mounted on unmanned aerial vehicles (UAVs) and automobiles to measure the shape of the earth's surface and features. For example, in an MMS (Mobile Mapping System) using an automobile, a GPS receiver, video camera, IMU, and laser scanner are mounted on the roof of the automobile, and measurements are taken while the automobile is traveling along roads. The video camera can be either analog or digital.

Three-dimensional point cloud data showing the earth's surface and the outlines of features can be obtained through such measurements. Each piece of three-dimensional point cloud data is given a position in absolute coordinates (plane rectangular coordinates, latitude, longitude, altitude, etc.) based on the position obtained by GPS and the scanning direction and distance of the laser scanner. In addition, the earth's surface and features are captured by a video camera to generate video data (still image data). This video data is linked to and recorded with the capturing position, which changes from moment to moment.

Obtaining 3D point cloud data allows us to understand the current state of the earth's surface and features, which can be useful for maintenance, map creation, etc.

The inventors propose assigning attribute data to features in 3D point cloud data, including data indicating their area in absolute coordinates and names.

However, while the above attribute data has led to progress in standardization and the accumulation of abundant data, there is a demand for a system that can efficiently manage features when using the attribute data for maintenance, etc. For example, there is a demand for a system that can appropriately manage photographs of the exterior of each feature.

The purpose of this invention is to provide a system that can solve the above problems and efficiently manage features.

The independently applicable features of this invention are listed below:

(1)(2)(3)The feature management system according to the present invention is a feature management system including a server device and a terminal device capable of communicating with the server device, the server device including a recording unit that records three-dimensional point cloud data including features whose positions are defined, a receiving means that receives captured images including the features and the captured positions transmitted from the terminal device, and an image/point cloud data matching means that projects the three-dimensional point cloud data onto a plane from different angles with the received captured positions as the center to generate two-dimensional projected images from a plurality of angles, and matches the captured images of the features with the three-dimensional point cloud data of the features based on a comparison between the captured images including the received features and each of the generated two-dimensional projected images, and the terminal device includes an image transmitting means that transmits the captured images including the features captured by a camera together with the captured positions to the feature management server device.

Therefore, photographs of the feature can be recorded in association with the 3D point cloud data of the feature.

(4) The feature management system according to the present invention is characterized in that the captured image/point cloud data correspondence means comprises: captured image outer shape generation means for indicating the outer shape of the feature in the captured image; two-dimensional projected image outer shape generation means for indicating the outer shape of the feature in each of the two-dimensional projected images; and means for comparing the outer shape of the feature in the captured image with the outer shape of the feature in each of the two-dimensional projected images, selecting a two-dimensional projected image whose outer shape matches, and corresponding the captured image of the feature with the three-dimensional point cloud data of the feature.

Therefore, by using an external shape that clearly changes depending on the angle, the matching process can be easily performed.

(5) The feature management system according to the present invention is characterized in that the recording unit of the server device records the outline area surrounding the outline of the feature and the feature name, the captured image/point cloud data correspondence means has a feature estimation means for extracting a feature portion from the captured image and estimating the feature name, projects three-dimensional point cloud data of a feature having the same feature name as the estimated feature name onto a plane from different angles with the received imaging position as the center to generate two-dimensional projected images from a plurality of angles, and associates the captured image of the feature with the three-dimensional point cloud data of the feature based on a comparison between the captured image of the received feature portion and each of the generated two-dimensional projection images.

Therefore, matching can be performed only on specified features, improving accuracy.

(6) The feature management system according to the present invention is characterized in that the recording unit of the server device records the outline area surrounding the outline of the feature and the feature name, the image transmission means of the terminal device receives input from an operator and assigns the boundary box and feature name of the feature in the captured image and transmits it to the server device, and the captured image/point cloud data correspondence means projects three-dimensional point cloud data of a feature having the same feature name as the transmitted feature name onto a plane from different angles, centered on the received imaging position, to generate two-dimensional projected images from a plurality of angles, and corresponds the captured image of the feature to the three-dimensional point cloud data of the feature based on a comparison between the captured image of the received feature portion and each of the generated two-dimensional projection images.

Therefore, matching can be performed only on specified features, improving accuracy.

(7) The feature management system according to the present invention is characterized in that the captured image/point cloud data correspondence means generates a plurality of virtual capture positions in the vicinity of the capture position, and generates a two-dimensional projected image by projecting the three-dimensional point cloud data onto a plane based on each virtual capture position.

This allows the imaging position to be identified more accurately.

(8) The feature management system according to the present invention is characterized in that the captured image/point cloud data correspondence means also identifies the imaging position and imaging direction for the feature.

Therefore, it is possible to identify not only the imaging position but also the imaging direction and utilize this information.

(9) The feature management system according to the present invention is characterized in that the captured images of the features transmitted from the terminal device are also accompanied by the capturing direction, and the captured image/point cloud data correspondence means narrows down the angles for generating the two-dimensional projection image based on the capturing direction.

This allows for faster matching processes.

(10) The feature management system of the present invention is characterized in that the three-dimensional point cloud data and the outer area of the attribute data are represented in absolute coordinates.

Therefore, attribute data and three-dimensional point cloud data can be recorded independently, and attribute data generated from three-dimensional point cloud data can be used for other three-dimensional point cloud data.

(11) The feature management system according to the present invention is characterized in that the feature extraction means projects the feature's outer shape area data or the feature's three-dimensional point cloud data onto a plane based on the past image capturing position and image capturing direction of the feature, generates a past two-dimensional projection image, and transmits this to a second terminal device, and the second terminal device displays the position two-dimensional projection image superimposed on the camera's viewfinder image when capturing an image with the camera.

This makes it easy to capture images of features from the same positions and angles as when they were captured previously.

(12) The feature management system according to the present invention is characterized in that the server device includes a point cloud data server device that records the three-dimensional point cloud data, and an attribute data server device that records attribute data including the outer region.

Therefore, it is possible to provide and utilize an attribute data server device that independently records attribute data.

(13) The feature management system of the present invention is characterized in that it projects the three-dimensional point cloud data onto a plane from different angles, centered on the imaging position when the image including the feature was captured, to generate two-dimensional projected images from a plurality of angles, and associates the captured image of the feature with the three-dimensional point cloud data of the feature based on a comparison between the captured image and each of the generated two-dimensional projected images.

Therefore, photographs of the feature can be recorded in association with the 3D point cloud data of the feature.

(14)(15)(16)The feature management system according to the present invention is characterized in that the server device includes a recording unit for recording three-dimensional point cloud data in which a first region and a second region having defined positions are distinguished, a receiving means for receiving an image including the first region and the second region and the image capturing position transmitted from the terminal device, an area estimation means for extracting the first region and the second region from other parts based on the image, and an image/point cloud data matching means for projecting the three-dimensional point cloud data in which the first region and the second region are distinguished from each other on a plane from different angles with the received image capturing position as a center to generate two-dimensional projected images from a plurality of angles, and for matching the captured image with the three-dimensional point cloud data based on a comparison between the captured image in which the first region and the second region are distinguished and each of the generated two-dimensional projected images, and the terminal device includes an image transmitting means for transmitting the captured image including the first region and the second region captured by a camera together with the image capturing position to the feature management server device.

Therefore, even if only a partial area of a feature is imaged rather than the entire feature, the imaging position and direction can be identified.

(17)(18)(19)The feature management system according to the present invention is characterized in that the server device includes a recording unit that records three-dimensional point cloud data including the feature and past imaging positions and imaging directions, and a past two-dimensional projection image generating means that projects the three-dimensional point cloud data of the feature or a three-dimensional model thereof onto a plane based on the past imaging positions and imaging directions to generate a past two-dimensional projection image and transmits it to the terminal device, and the terminal device includes a display means that displays the past two-dimensional projection image received from the server device on the display unit by superimposing it as a transparent image on the viewfinder image of a camera for imaging the feature.

Therefore, the imaging position can be adjusted while referring to previous images, and imaging can be performed in the same position and direction as when the previous images were captured.

(20) The feature management method according to the present invention is characterized in that it records three-dimensional point cloud data including features and past imaging positions and imaging directions, projects the three-dimensional point cloud data of the features or a three-dimensional model thereof onto a plane based on the past imaging positions and imaging directions to generate a past two-dimensional projection image, and displays the past two-dimensional projection image superimposed on the camera viewfinder image.

Therefore, the imaging position can be adjusted while referring to previous images, and imaging can be performed in the same position and direction as when the previous images were captured.

(21)(22) The feature management system according to the present invention is a feature management system including a server device and a mobile terminal device capable of communicating with the server device, the server device including a recording unit that records three-dimensional point cloud data including features and past imaging positions and imaging directions, and a transmission means that transmits the imaging positions and imaging directions to the terminal device, and the terminal device includes a display means that displays the received imaging positions and imaging directions on a map on a display unit.

This makes it easy to know the previous imaging position and imaging direction.

(23)(24)(25)The feature management system according to the present invention is a feature management system including a server device and a terminal device capable of communicating with the server device, the server device records three-dimensional point cloud data with reflection intensity including road display features and outline area data surrounding the outline of the road display features, the terminal device includes an outline area data acquisition means for acquiring the outline area data of the road display features from the server device, a three-dimensional point cloud data acquisition means for acquiring three-dimensional point cloud data corresponding to the outline area data of the road display features from the server device, and a display means for displaying the acquired outline area of the road display features and the three-dimensional point cloud data superimposed on a display unit.

This makes it easy to determine the degree of deterioration of road marking features.

(26) The feature management method according to the present invention is characterized in that it records three-dimensional point cloud data with reflection intensity including a road display feature and outline area data surrounding the outline of the road display feature, acquires the outline area data of the road display feature, acquires three-dimensional point cloud data corresponding to the outline area data of the road display feature, and displays the acquired outline area of the road display feature and the three-dimensional point cloud data superimposed on the display unit.

This makes it easy to determine the degree of deterioration of road marking features.

In this embodiment, the "receiving means" corresponds to the processing performed by the attribute data server device 60 in response to step S602.

In this embodiment, steps S701 to S707 correspond to the "captured image/point cloud data correspondence means."

In this embodiment, step S602 corresponds to the "image transmission means."

In this embodiment, step S651 corresponds to the "external area data acquisition means."

In this embodiment, step S652 corresponds to the "three-dimensional point cloud data acquisition means."

In this embodiment, step S653 corresponds to the "display means."

The term "program" refers not only to programs that can be executed directly by a CPU, but also to source-format programs, compressed programs, encrypted programs, etc.

FIG. 1 is a functional block diagram of a feature management system according to an embodiment of the present invention. This shows the system configuration of a feature management system. 2 shows the hardware configuration of the mobile terminal device 21. 2 shows the hardware configuration of the attribute server device 60. 13 is a flowchart of a terminal program. 1 is an example of three-dimensional point cloud data. FIG. 1 illustrates a process for detecting ground points. This is an example of design data for a feature. FIG. 1 is a diagram showing three-dimensional point cloud data superimposed on a plan view of a feature. This is an example of 3D point cloud data of geographical features. 13 is an example of attribute data items. 11 is a diagram for explaining a process of estimating a feature name based on three-dimensional point cloud data of the feature. FIG. 13 is a flowchart of a feature management process. 13 is a flowchart of a feature management process. 13 is a flowchart of a feature management process. 1 is a captured image of a feature captured by a user. FIG. 1 is a diagram showing a three-dimensional model of a feature generated based on three-dimensional point cloud data. This is an example of a two-dimensional projection image generated by projecting at different angles. This is a contour image obtained by extracting the contours of features from a captured image. This is a captured image of a feature cut out based on the contour line of the feature in a two-dimensional projected image. 1 is a diagram showing an imaging position P and virtual positions P1, P2 . . . P23, and P24. FIG. 13 is a diagram showing a captured image in which features are marked by a user. This is an example of the previous image being displayed overlaid on the camera's viewfinder screen. FIG. 11 is a functional block diagram of a feature management system according to a second embodiment. 13 is a flowchart of a feature management process. FIG. 13 is a diagram showing a state in which luminance-added three-dimensional point cloud data of the center line and an outline region are displayed in an overlapping manner. FIG. 13 is a functional block diagram of a point cloud data management system according to a third embodiment of the present invention. This shows the system configuration of a point cloud data management system. This shows the hardware configuration of the terminal device. 1 is a flowchart for acquiring 3D point cloud data of features and generating a 3D model. 1 is a flowchart for acquiring 3D point cloud data of features and generating a 3D model. 13 is a display example of the outer shape of a feature. FIG. 33A is first 3D point cloud data of a feature from a first direction, and FIG. 33B is second 3D point cloud data of a feature from a second direction. This is a composite 3D point cloud data. Fig. 35A shows the generated three-dimensional model, and Fig. 35B shows an example of the measurement direction. FIG. 36A shows the first three-dimensional point cloud data, FIG. 36B shows the second three-dimensional point cloud data, and FIG. 36C shows the composite three-dimensional point cloud data. FIG. 13 is a functional block diagram of a point cloud data management system according to a fourth embodiment. 13 is a flowchart for acquiring a change over time. 13 is a flowchart for acquiring a change over time. This is an example of how features change over time. 13 is an example of a screen prompting the user to select three-dimensional point cloud data with different dates and times. FIG. 13 is a functional block diagram of a feature search system according to a fifth embodiment. This is the system configuration of the feature search system. 1 is a flowchart of a feature search. FIG. 2 is a diagram showing a configuration of video data. FIG. 13 is a flowchart of video extraction. FIG. 2 is a diagram showing the relationship between an imaging trajectory 200 and a feature 106 at each time. FIG. 13 is a diagram showing an image in which search target features are marked.

1. First embodiment
1 shows the overall configuration of a feature management system according to an embodiment of the present invention. A recording unit 214 of a server device 210 records three-dimensional point cloud data 222 based on absolute coordinates including features.

The mobile terminal device 240 is equipped with a camera 244. Images of features captured by the camera 244 are transmitted to the server device 210 by the image transmission means 242. At this time, if the camera 244 has an image capture position addition function, the image capture position (position based on absolute coordinates) is also transmitted to the server device 210. If the camera 244 does not have such a function, the image capture position based on GPS or the like is added and transmitted to the server device 210.

The receiving means 216 of the server device 210 receives the captured image and the imaging position. The two-dimensional projection image generating means 218 generates two-dimensional projection images for multiple angles when the surrounding three-dimensional point cloud data is viewed from the imaging position based on the imaging position. The comparing means 220 selects, from the multiple generated two-dimensional projection images, the two-dimensional projection image that best matches the received captured image.

As a result, the captured image of the feature is recorded in association with the feature in the selected two-dimensional projected image. In this way, the captured image showing the external appearance or the like can be recorded in association with the three-dimensional point cloud data.

2 shows the system configuration of a point cloud data management system according to an embodiment. In this embodiment, the point cloud server devices 40a, 40b, . . . 40n and the attribute data server device 60 constitute a server device.

Point cloud server devices 40a, 40b, ... 40n are provided on the Internet. An attribute data server device 60 is also provided. Terminal devices 20a ... 20n and mobile terminal devices 20a ... 20n are provided on these server devices 40a, 40b ... 40n, 60 so as to be able to communicate via the Internet.

The hardware configuration of the mobile terminal device 21 is shown in FIG. 3. A memory 82, a touch display 84, a non-volatile memory 86, a communication circuit 92, a camera 88, and a GPS receiver 98 are connected to a CPU (or a GPU, etc.; the same applies below) 80. The communication circuit 92 is for connecting to the Internet. The mobile terminal device 21 may be a smartphone, a tablet, a wearable computer, or the like.

The non-volatile memory 86 stores an operating system 94 and a mobile terminal program 96. The mobile terminal program 96 works in conjunction with the operating system 94 to perform its functions.

The hardware configuration of the attribute data server device 60 is shown in Figure 4. The CPU 81 is connected to a memory 83, a display 85, a hard disk 87, a DVD-ROM drive 89, a keyboard/mouse 91, and a communication circuit 93. The communication circuit 93 is for connecting to the Internet.

The hard disk 87 stores an operating system 95 and an attribute server program 97. The attribute server program 97 performs its functions in cooperation with the operating system 95. These programs were recorded on a DVD-ROM 99 and installed onto the hard disk 87 via the DVD-ROM drive 89.

The terminal devices 20a to 20n and the point cloud server devices 40a, 40b to 40n have the same hardware configuration. However, in the terminal device 20a, a terminal program is recorded instead of the attribute server program 97. Also, in the point cloud server devices 40a, 40b to 40n, a point cloud server program is recorded instead of the attribute server program 97.

1.3 Disclosure of point cloud data The surveying company generates 3D point cloud data and videos of roads and features using an MMS or the like installed in a vehicle, etc. The generated 3D point cloud data is recorded on a recording medium such as a DVD-ROM, and is imported into the hard disk 87 of the terminal device 20.

Here, features include, for example, road design standard information (mileposts, road centerlines, measurement points, etc.), road surface features (roadway sections, sidewalks, trackbeds, etc.), features that share an area with the road surface (stop lines, dividing lines, pedestrian crossings, road markings, etc.), features other than the road surface (signposts, lighting poles, traffic lights, footbridges, etc.), and features that indicate the road surface (retaining walls, bridges, slopes, tunnels, sheds, etc.). Note that this is not limited to road features.

The surveying company starts the terminal program and generates attribute data based on the recorded 3D point cloud data. Figure 5 shows a flowchart of the attribute data generation process in the terminal program 96.

First, the CPU 81 of the terminal device 20a reads out the recorded three-dimensional point cloud data (step S1). An example of the three-dimensional point cloud data is shown in FIG. 6. In this embodiment, the three-dimensional point cloud data is generated by measuring the color of the object to be measured and the reflection intensity of the laser. The three-dimensional point cloud data also records information on the measurement method and the measurement equipment.

The CPU 80 groups together points within a predetermined distance in the three-dimensional point cloud data into clusters. This results in the formation of clusters of various sizes. The CPU 81 then removes clusters that are below a predetermined volume, a predetermined number of point clouds, or a predetermined point cloud density as noise (step S2).

Next, the CPU 81 extracts ground points (ground surface) using a Cloth Simulation method or the like (step S2). In extracting ground points, the CPU 80 first inverts the elevation values of the three-dimensional point cloud data. For example, if there is three-dimensional point cloud data of a cross section as shown in FIG. 7A (the point cloud is shown as a line in the figure), inverted three-dimensional point cloud data as shown in FIG. 7B can be obtained.

Next, the CPU 81 performs a simulation on the inverted three-dimensional point cloud data as if a cloth were draped over it from above. In FIG. 7C, the simulated cloth is shown by a dashed line. The CPU 81 then extracts the three-dimensional point cloud data that the simulated cloth contacts as ground points. Next, the CPU 81 re-inverts the elevation values to obtain ground points as shown in FIG. 7D.

The ground points extracted in this way are generally accurate, but as shown in Figure 7D, they may contain some features in the vicinity 100 where the features are present. Therefore, the normal direction of the line formed by each extracted ground point is calculated, and the part where the normal line forms a certain angle or more (e.g., 30 degrees or more) with respect to the vertical direction is excluded from the ground points. This makes it possible to obtain ground points such as those shown in Figure 7E.

In this embodiment, Cloth Simulation is used to extract ground points, but ground points may be extracted using other methods, such as the lowest point extraction method.

After extracting the ground points in the above manner, the CPU 81 removes the ground points from the three-dimensional point cloud data. This results in three-dimensional point cloud data of only the features that exist on the ground (step S4).

Next, the CPU 81 reads out design data that indicates the planar arrangement of the features recorded on the hard disk 87. This design data is a drawing of the features at the time of design, obtained by measuring the above-mentioned three-dimensional point cloud data. An example of a design drawing is shown in FIG. 8. The CPU 80 extracts the planar shape of each feature based on this design data (step S5). In this embodiment, the planar shape is extracted as a rectangle that contains the planar outer shape of the feature. Note that the planar outer shape of each feature may also be extracted as the planar shape as it is.

Next, the CPU 81 overlays the three-dimensional point cloud data of only the feature on this planar shape. The absolute coordinates of the design data on which the planar shape is based are matched with the absolute coordinates of the three-dimensional point cloud data to perform the overlay. At this time, the height position of the planar shape is matched with the height (altitude) of the ground point (ground surface) of the three-dimensional point cloud data. The matched state is shown in FIG. 9. In FIG. 9, only the footbridge portion of the features in FIG. 7 and FIG. 8 is shown. The frame line 300 shows the rectangular planar shape that surrounds the planar outer shape of the footbridge (feature), and it can be seen that the position of this planar shape 300 and the three-dimensional point cloud data of the footbridge (feature) match. Note that the planar outer shape is originally represented as a plane, but in FIG. 9, it is shown with a height for ease of understanding.

For each feature, the CPU 81 selects the one with the highest elevation from the three-dimensional point cloud data corresponding to the planar shape of the feature. This height is applied to the planar shape to determine the area of the feature. This makes it possible to identify a three-dimensional shape (rectangular prism) that surrounds the feature, as shown in FIG. 10. The CPU 81 records the three-dimensional shape generated in this way as outline area data. In this embodiment, the outline area data is recorded based on the planar shape and the position and height based on the absolute coordinates of the horizontal azimuth and height.

Next, the CPU 81 displays the area of each feature on the display 84 by overlaying it on the three-dimensional point cloud data. The operator looks at this screen and selects the area of each feature with the mouse 91. This displays an input screen for attribute data such as the feature name, and the operator inputs the data. In addition, based on the coordinates of the vertices of the three-dimensional shape of the outer shape area data, the coordinates of the center position (or a position representative of the feature may be used) are calculated and set as the coordinate position of the feature. In this way, attribute data can be generated and recorded. Figure 11 shows the items of attribute data in this embodiment. The creation date and time is the creation date and time (measurement date and time) of the three-dimensional point cloud data. The feature name is the name of the feature. The feature area is the outer shape data that indicates the outer shape of the feature. The feature position is the center position calculated above. In this embodiment, the three-dimensional point cloud data is recorded in XML format.

In the above, the operator inputs the feature name, but the feature name may be estimated based on the three-dimensional point cloud data. For example, as shown in FIG. 12, the three-dimensional point cloud data 6 of the feature is projected onto a plane at a different angle, and the projected image and the feature name are used as learning data to train a deep learning program (or other machine learning programs may be used). Learning is performed on various features to obtain a trained feature estimation program. The feature estimation program may be provided with three-dimensional point cloud data for which the feature is to be estimated, and the feature name may be automatically estimated and assigned. Alternatively, the estimated feature name may be displayed to the operator for confirmation.

In addition, since there is a possibility that the three-dimensional point cloud data 6 and the outer region are misaligned, they may be displayed on the display 85, and the operator may align them using a mouse or the like so that their positions match.

This is how you can obtain attribute data.

In the above example, the outline area of the feature is determined by assigning height to the planar data. However, if there is no planar data such as design data, the outline area may be determined from the 3D point cloud data itself.

For example, the outline region is determined as follows: The three-dimensional space is divided into small cubic grids, and if there are points in adjacent grids above, below, left, right, or diagonally, these are merged together. For example, a spatial labeling technique using connected components can be used.

For each area organized as a grid containing points, areas with a certain volume, a certain number of points, or a certain density of points are removed as noise. The remaining areas correspond to features. For each area, a rectangular prism is set that surrounds the area. This rectangular prism is the outline of the feature.

In this way, attribute data corresponding to the three-dimensional point cloud data can be generated on the terminal device 20. The surveying business delivers this three-dimensional point cloud data and attribute data to the delivery destination.

Furthermore, at the discretion of the surveying company, only the attribute data can be uploaded to the attribute data server device 60. This allows companies that require three-dimensional point cloud data to access the attribute data server device 60 and refer to the attribute data. If information about the company providing the corresponding three-dimensional point cloud data is recorded in the attribute data, companies that require three-dimensional point cloud data can use this information to request the surveying company to obtain the three-dimensional point cloud data.

The three-dimensional point cloud data may also be uploaded to the point cloud server device 40. If the requester of the three-dimensional point cloud data is a local government or the like, they may request that it be uploaded to a specific point cloud server device 40. This makes the three-dimensional point cloud data and attribute data publicly available on the Internet.

Furthermore, the three-dimensional point cloud data and the outer area data of the attribute data are shown in absolute coordinates. Therefore, when there are multiple three-dimensional point cloud data for the same feature uploaded to different point cloud server devices by different businesses, these multiple three-dimensional point cloud data can be acquired as long as at least one attribute data is uploaded. For example, when there are three-dimensional point cloud data for the same feature from different directions by different businesses, more complete three-dimensional point cloud data can be obtained by acquiring both.

1.4 Feature Management Processing Figure 13 shows a flowchart of the feature management processing. For example, a user (such as a person in charge of managing a feature) goes near a damaged feature and takes an image of the feature with the camera 88 of the mobile terminal device 21 (step S601). The CPU 80 of the mobile terminal device 21 (hereinafter, sometimes abbreviated as the mobile terminal device 21) transmits the captured image thus generated to the attribute data server device 60 in response to the user's operation (step S602). At this time, the mobile terminal device 21 also transmits the position (imaging position) acquired by the GPS receiver 98 at the time of imaging. An example of a captured image is shown in Figure 16.

The CPU 81 of the attribute data server device 60 (hereinafter sometimes abbreviated as the attribute data server device 60) receives this and acquires attribute data of features in the vicinity of the imaging position (step S701). Since the attribute data records the type of feature (feature name) and the location of the feature, features within a specified distance from the imaging position are extracted.

Next, based on the extracted contour area of the feature, the attribute data server device 60 requests each point cloud server device 40a, 40b, ... 40n to transmit the three-dimensional point cloud data (i.e., three-dimensional point cloud data of the feature) contained in the contour area if it has been recorded (step S702).

Among the point cloud server devices 40a, 40b, ... 40n, the point cloud server device 40 that has recorded and stored the corresponding three-dimensional point cloud data responds by returning the three-dimensional point cloud data of the feature to the attribute data server device 60 (step S801).

The attribute data server device 60 generates a three-dimensional model (a three-dimensional model specified by absolute coordinates) that represents the outer shape based on the received three-dimensional point cloud data of the feature (step S703). Note that this three-dimensional model may be generated and recorded in advance. An example of a three-dimensional model generated for each feature is shown in FIG. 17.

Next, for the 3D model generated for each feature, a 2D projection image is generated while changing the angle, centered on the imaging position PH shown in FIG. 17 (step S704). Examples of the generated 2D projection images are shown in FIGS. 18A, 18B, and 18C. Although FIG. 18 shows only 2D projection images from three angles, 2D projection images from other angles are generated in the same manner.

The attribute data server device 60 then generates a contour image of the feature captured in the captured image. The process of extracting the contour of a specific object (here, the feature) captured in the captured image can be performed, for example, by Mask R-CNN (https://github.com/matterport/Mask_RCNN). Figure 19 shows the contour image.

The attribute data server device 60 compares the contoured captured image of FIG. 19 with each of the two-dimensional projected images shown in FIG. 18, and selects the one that matches best (step S705). For this matching, for example, SIFT, SURF, KAZE, or A-KAZE can be used. For example, it is assumed that the two-dimensional projected image of FIG. 18B is selected. Through this matching, the imaging direction can also be obtained based on the angle at which FIG. 18B was generated.

Next, the attribute data server device 60 extracts the outline of the feature in the selected two-dimensional projection image. Furthermore, the captured image is cropped based on this outline to obtain an image of only the feature (step S706). The resulting cropped image of the feature is shown in Figure 20.

The attribute data server device 60 records the feature image, the imaging position, and the imaging direction in association with the attribute data of the feature (step S707).

Therefore, the attribute data server device 60 can be accessed from each of the terminal devices 20a to 20n and each of the mobile terminal devices 21a to 21n to view and obtain captured images of each feature. This makes it easy to manage each feature. Also, the person taking the images at the site does not need to link the images to the three-dimensional point cloud data of the features himself, and the correspondence can be performed automatically.

1.5 Other
(1) In the above embodiment, when generating a two-dimensional projection image, the imaging position received from the mobile terminal device 21 is used as is. However, if there is a problem with the positional accuracy, as shown in Fig. 21, virtual positions P1, P2...P23, P24 may be set around the received imaging position P, and two-dimensional projection images may be generated with each of them as the center.

The two-dimensional projection images generated in this way are matched with the captured images, and the two-dimensional projection image that best matches is selected. The virtual position corresponding to the selected two-dimensional projection image is recorded as the corrected captured position.

In addition to the planar position (latitude and longitude), the height (altitude) may also be changed up or down relative to the received height to set the virtual position.

(2) In the above embodiment, matching is performed by comparing the contours of features in the captured image with the contours of features in the two-dimensional projected image (three-dimensional model). However, it is also possible to project three-dimensional point cloud data into two dimensions and compare the point cloud data with the captured image. In this case, if the captured image is in color, the two-dimensionally projected point cloud data can also be made color based on the color data at the time of measurement, and matching including color can be performed to improve the accuracy of determining the degree of match.

(3) In the above embodiment, two-dimensional projection images are generated for all features within a specified range from the imaging position, and are then matched with the entire captured image. However, if substantially only one feature is captured in the captured image, it is also possible to identify the name of the feature in the captured image, generate a two-dimensional projection image for only the three-dimensional point cloud data for the identified feature name, and match it with the captured image of the feature.

In this case, for example, as shown in FIG. 22, the user may operate the mobile terminal device 21 to mark the portion of the feature that catches the user's attention in the captured image, and transmit the mark to the attribute data server device 60. At this time, the user operates the mobile terminal device 21 to input the feature name (preferably selected from a pull-down menu of predetermined feature names), and transmits the name as well.

The attribute server device 60 selects the feature with the name sent from among the features within a specified range from the imaging position, and generates a two-dimensional projection image. At this time, a two-dimensional projection image from a different angle is generated for each feature.

In matching, the captured image of the feature selected by the user is compared with the two-dimensional projected image of each feature. The captured image of the feature is recorded in association with the two-dimensional projected image of the feature that best matches.

The matching can be done using a three-dimensional model or point cloud data.

In the above, the user inputs the feature name. However, the mobile terminal device 21 or the attribute data server device 60 may identify objects in the captured image and determine the feature name. Using a learning model such as deep learning that learns the image and feature name of each feature as learning data, the features in the image can be extracted and their names obtained (for example, programs such as YOLO and Mask R-CNN can be used). In this case, it is preferable that the user selects a feature from the captured image and the mobile terminal device 21 or the attribute data server device 60 automatically determines the feature name of the selected feature.

(4) In the above embodiment, a two-dimensional projection image is generated for features within a specified range from the imaging position, and then matched with the captured image. Only image information is used in the matching.

However, the user may input the names of the features in the captured image, or the mobile terminal device 21 or the attribute data server device 60 may automatically determine the names (using YOLO, etc., as described above) and perform matching including the names of the features.

In addition, when multiple features are present in a captured image, the names of each feature are identified from the image, while
The names of each feature may be obtained from the attribute data, and matching may be performed including the positional relationships (order on the image) of these feature names.

(5) In the above embodiment, the three-dimensional point cloud data and attribute data are expressed using absolute coordinates. However, as long as the three-dimensional point cloud data and attribute data correspond to each other, absolute coordinates are not necessarily required.

(6) In the above embodiment, the accuracy of matching improves if there are many features around the imaging position. However, even if only the road and the slope are captured in the captured image, the imaging position can be identified.

For example, it can be used when an accident occurs on a mountain road where there are no signs or other features, and the accident site can be identified based on a photograph of the scene. In this case, the captured image is separated into the road surface and slopes using a model trained by deep learning such as SegNet, an image segmentation method. Meanwhile, a two-dimensional planar image is generated that separates the road surface and slopes from each position using the three-dimensional point cloud data. The point where the two match best can be determined to be the capture location (accident site).

In addition, when a feature has multiple similar structures (for example, bridge girders), the imaged location can be identified by setting the above-mentioned virtual imaging position and imaging direction for each location, generating a two-dimensional projection image, and performing matching. This makes it possible to identify which location (which of the multiple bridge girders) was imaged.

(7) In the above embodiment, the mobile terminal device 21 transmits the captured image and the capture position to the attribute data server device 60. However, the mobile terminal device 21 may be provided with a geomagnetic sensor (to obtain orientation) and a gyro sensor (to obtain pitch, roll, and other inclinations) so that the capture direction may also be transmitted. Based on the received capture direction, the attribute data server device 60 generates a two-dimensional planar image only for the angles before and after the capture direction. This can improve matching accuracy.

(8) In the above embodiment, past history was not used when capturing images of features with a camera. However, when capturing images of features with a camera, the capturing position and capturing direction may be transmitted to the attribute data server device 60, and if a result of capturing an image from a similar direction in the vicinity in the past has been saved, this may be acquired. The attribute data server device 60 generates a two-dimensional projection image based on the past capturing position and capturing direction, and transmits this to the mobile terminal device 21. The mobile terminal device 21 that receives the past two-dimensional projection image displays the past image IM superimposed as a transparent image on the camera's finder screen, as shown in FIG. 23. Although FIG. 23 shows only past images IM of some features, the entire past image may be displayed.

This makes it easy to match the position and direction when it is necessary to capture an image at the same location as one captured previously. For example, this is useful when observing changes over time of a particular feature.

In addition, instead of a past two-dimensional projection image, a past captured image may be displayed superimposed as a transparent image.

Furthermore, instead of or in addition to displaying past images on top of each other, the imaging position and imaging direction at the time of past imaging may be shown on the map. In this case, by also showing the current imaging position and imaging direction, it becomes easier to capture images that match the past imaging position and imaging direction.

(9) In the above embodiment, features in the captured image are identified and matching is performed based on their contours and the contours of the three-dimensional point cloud data. However, it is also possible to extract all contours (edges) of features and other than features from the captured image and match these with the contours of the three-dimensional point cloud data. For example, a Prewitt filter, a Sobel filter, a Laplacian filter, a LoG filter, a DoG filter, a Canny method, a Hough transform, etc. can be used to extract edges from the captured image.

(10) In the above embodiment, the first point cloud server device 40a and the second point cloud server device 40b are configured as separate server devices, but they may be configured as a single server device.

(11) In the above embodiment, the first point cloud server device 40a, the second point cloud server device 40b, and the attribute data server device 60 are constructed as separate server devices, but they may also be constructed as a single server device.

(12) In the above embodiment, steps S702, S703, S704, S705, and S706 are performed by the attribute data server device. However, this may be performed by a mobile terminal device or a terminal device, and the extracted images of the features may be transmitted to the attribute data server device for recording.

(13) In the above embodiment, the mobile terminal device captures images of features and transmits the captured images. However, images captured by a camera or the like may be imported into a terminal device such as a PC and transmitted.

(14) The above-described embodiments and variations can be implemented in combination with other embodiments as long as this does not go against the essence of the invention.

2. Second embodiment
24 shows a functional block diagram of a feature management system according to the second embodiment. In this embodiment, an attribute server device 211 and a three-dimensional point cloud server device 213 constitute a server device.

The outer region data acquisition means 262 of the terminal device 260 acquires the outer region data 224 of the road markings (land features) from the attribute server device 211. It is preferable to use the outer region data at the time of installation of road markings such as center lines.

The three-dimensional point cloud data acquisition means 264 acquires three-dimensional point cloud data 222 of road objects included in the outer region based on the outer region data 224. This three-dimensional point cloud data 222 also includes the brightness of each point.

The display means 266 displays the outline region based on the acquired outline region data and each point of the three-dimensional point cloud data with brightness (higher brightness means brighter, and lower brightness means darker). This allows the condition of the road marking at the time of installation to be displayed as the outline region, and the current condition to be displayed as the brightness of the three-dimensional point cloud data. This allows the user to know the necessity of repairs to the road marking, the repair locations, etc.

2.2 System Configuration The system configuration is the same as that shown in Fig. 2 in the first embodiment. The hardware configurations of the point cloud server device 40, the attribute data server device 60, the terminal device 20, and the mobile terminal device 21 are also the same as those in the first embodiment.

2.3 Feature Management Processing A flowchart of the feature management processing is shown in Figure 25. In the figure, the terminal device is shown as a flowchart of the terminal program, the attribute data server device is shown as a flowchart of the attribute data server program, and the point cloud server device is shown as a flowchart of the point cloud server program.

First, the CPU 81 of the terminal device 20b (hereinafter sometimes abbreviated as terminal device 20b) accesses the attribute data server device 60 and specifies the location where the desired road object is located (step S651). The location is specified by specifying a range using, for example, latitude and longitude. An address may be input and converted into a latitude and longitude range.

In response to this, the CPU of the attribute server device 60 (hereinafter sometimes abbreviated as the attribute server device) searches the attribute data and extracts the outer area data of the road markings within the specified location (step S751). Here, a road marking is one of the features, and is an identification mark displayed on the road surface. For example, it is a center line, a crosswalk, etc.

The attribute server device 60 transmits the extracted outer region data (planar shape data and its position and height in absolute coordinates), feature names (center line, pedestrian crossing, etc.) to the terminal device 20b. If multiple outer region data are extracted, these multiple outer region data are transmitted.

The terminal device 20b accesses each point cloud server device 40a...40n and requests that the three-dimensional point cloud data of the received outer region data be returned (step S652). If multiple outer region data exist for the same road display object, the terminal device 20b may display the measurement date and time of the outer region data and allow the user to select. The user selects the oldest outer region data or the outer region data from when the road display object was installed. The terminal device 20b requests three-dimensional point cloud data based on the selected outer region data.

In response to this, among the point cloud server devices 40a...40n, the point cloud server device that has recorded the three-dimensional point cloud data included in the external region data transmits the three-dimensional point cloud data to the terminal device 20b in association with each external region data (step S851).

The terminal device 20b displays the received outline region data by overlaying the brightness of the three-dimensional point cloud data (step S653). An example of this display is shown in FIG. 26. Here, a road display object of a center line is shown as an example. The outline region 600 is displayed based on the outline region data. The high brightness region 602 is an area displayed by the three-dimensional point cloud data with high brightness. Although the three-dimensional point cloud data is a point, in FIG. 26 it is shown as an area since it is a collection.

If the entire outline region 600 is filled with high-brightness regions 602, it can be determined that the center line is sound and has no peeling or other defects. If there is peeling or damage on the center line, the reflection intensity of the laser in that area will be low. Therefore, as shown in Figure 26, low-brightness regions 604 are displayed within the outline region 600. This makes it possible to know the center line that needs repair and the repair locations.

2.4 Other
(1) In the above embodiment, a rectangular feature (road object) having a planar area has been described as the target. However, the same processing can be performed on a rectangular feature having a three-dimensional area, the surface of which has a higher reflectance than the other parts.

(2) The above embodiment is directed to rectangular road objects. However, if a shape that conforms to the road object (for example, the shape of characters displayed on the road) is recorded as the outer shape data in the outer shape area data, it can also be applied to road objects of such shapes.

(3) In the above embodiment, the brightness of the three-dimensional point cloud data is compared with the external shape. However, the three-dimensional point cloud data itself (without brightness values) may be displayed and compared with the external shape.

(4) In the above embodiment, the first point cloud server device 40a and the second point cloud server device 40b are configured as separate server devices, but they may be configured as a single server device.

(5) In the above embodiment, the first point cloud server device 40a, the second point cloud server device 40b, and the attribute data server device 60 are constructed as separate server devices, but they may also be constructed as a single server device.

(6) The above-described embodiments and variations can be implemented in combination with other embodiments as long as this does not go against the essence of the invention.

3. Third embodiment
3.1 Overall Configuration Fig. 27 shows the overall configuration of a point cloud data management system according to a third embodiment of the present invention. A first three-dimensional point cloud data file from a first direction based on absolute coordinates including features is recorded in the first point cloud server device 40a. A second three-dimensional point cloud data file from a second direction based on absolute coordinates including features is recorded in the second point cloud server device 40b. An outer shape area data based on absolute coordinates that surrounds the outer shape of features is recorded in the attribute data server device 60.

The terminal device 20 is capable of communicating with these server devices. The three-dimensional point cloud data acquisition means 22 of the terminal device 20 acquires the contour area data of the desired feature from the attribute data server device 60. The three-dimensional point cloud data acquisition means 22 accesses the first point cloud server device 40a based on the acquired contour area data of the feature, and acquires the three-dimensional point cloud data included in the area as the first three-dimensional point cloud data. Furthermore, the three-dimensional point cloud data acquisition means 22 accesses the first point cloud server device 40b based on the acquired contour area data of the feature, and acquires the three-dimensional point cloud data included in the area as the second three-dimensional point cloud data.

The three-dimensional model construction means 24 constructs the three-dimensional shape of the feature based on the first three-dimensional point cloud data and the second three-dimensional point cloud data acquired above. In this way, by utilizing the fact that the outer shape area data is assigned absolute coordinates, it is possible to collect three-dimensional point cloud data from different directions of the target feature that is recorded in a dispersed manner on the Internet, etc. Therefore, it is possible to form a more complete three-dimensional model of the feature.

3.2 System Configuration Fig. 28 shows the system configuration of a point cloud data management system according to the third embodiment. Point cloud server devices 40a, 40b, ..., 40n are provided on the Internet. An attribute data server device 60 is also provided. These server devices 40a, 40b, ..., 40n, 60 are provided with terminal devices 20a, 20b, ..., 20n that are capable of communicating via the Internet.

The hardware configuration of the terminal device 20 is shown in FIG. 29. The CPU 80 is connected to a memory 82, a display 84, a hard disk 86, a DVD-ROM drive 88, a keyboard/mouse 90, and a communication circuit 92. The communication circuit 92 is for connecting to the Internet.

The hard disk 86 stores an operating system 94 and a terminal program 96. The terminal program 96 works in conjunction with the operating system 94 to perform its functions. These programs were recorded on a DVD-ROM 98 and installed onto the hard disk 86 via the DVD-ROM drive 88.

The point cloud server device 40 and the attribute data server device 60 have the same hardware configuration. However, in the point cloud server device 40, a point cloud server program is recorded instead of the terminal program 96. In addition, in the attribute data server device 60, an attribute data server program is recorded instead of the terminal program 96.

3.3 Three-dimensional Model Construction Process Next, a process for constructing a three-dimensional model based on a plurality of three-dimensional point cloud data published as described above will be described.

The following describes how to build a 3D model of a specific feature.

Figures 30 and 31 show flowcharts for constructing a three-dimensional model. In the figures, the terminal device is shown as a flowchart for the terminal program, the attribute data server device is shown as a flowchart for the attribute data server program, and the point cloud server device is shown as a flowchart for the point cloud server program.

First, the CPU 80 of the terminal device 20b (hereinafter sometimes abbreviated as terminal device 20b) accesses the attribute data server 60 and specifies the location where the desired feature is located (step S11). The location is specified by specifying a range using, for example, latitude and longitude. An address or place name may be entered (or searched), and this may be converted into a latitude and longitude range. Also, an area clicked or selected on the map may be converted into a latitude and longitude range.

In response to this, the CPU of the attribute server device 60 (hereinafter sometimes abbreviated as the attribute server device) searches the attribute data and extracts the contour area data within the specified location (which may include the vicinity outside the location) (step S21). The attribute server device 60 transmits the extracted contour area data (planar shape data and its position and height in absolute coordinates), feature names, etc. to the terminal device 20b.

The terminal device 20b displays the outline area and the feature name on the display 84 based on the received outline area data (step S12). An example of the display is shown in FIG. 32. The outline areas 102-112 of each feature are shown, and the feature name is shown in their vicinity.

The user operates the mouse 90 of the terminal device 20b to select the outline region of the desired feature. Here, the explanation will proceed assuming that the outline region 110 of the building has been selected. In response to this selection, the terminal device 20b specifies the outline region data of the outline region 110 of the building and transmits a request for three-dimensional point cloud data to each of the point cloud server devices 40a to 40n (step S13).

Each point cloud server device 40a-40n that receives this request determines whether it holds the three-dimensional point cloud data for the outer region 110, and if so, returns it. Here, the explanation will proceed assuming that two of the point cloud server devices 40a-40n, point cloud server devices 40a and 40b, hold the three-dimensional point cloud data for the outer region 110 of the building.

The CPU of the point cloud server device 40a (hereinafter sometimes abbreviated as point cloud server device 40a) returns the three-dimensional point cloud data of the exterior area 110 of the building to the terminal device 20b (step S31).

Similarly, the CPU of the point cloud server device 40b (hereinafter sometimes abbreviated as point cloud server device 40b) returns the three-dimensional point cloud data of the exterior area 110 of the building to the terminal device 20b (step S41).

The point cloud server devices 40a and 40b transmit three-dimensional point cloud data of an area that is wider than the specified outer area 110 to take into account measurement errors. Measurement errors differ depending on the measurement method and equipment, and therefore differ depending on the measurement method and equipment used to measure the three-dimensional point cloud data. Therefore, the above-mentioned wider area differs depending on the measurement method and equipment.

The terminal device 20b receives three-dimensional point cloud data (first three-dimensional point cloud data) from the point cloud server device 40a and three-dimensional point cloud data (second three-dimensional point cloud data) from the point cloud server device 40b. An example of the first three-dimensional point cloud data is shown in FIG. 33A, and an example of the second three-dimensional point cloud data is shown in FIG. 33B. As shown in FIG. 33A and FIG. 33B, the first three-dimensional point cloud data was measured from the front side of the rectangular parallelepiped facing the paper. The second three-dimensional point cloud data was measured from the rear side of the rectangular parallelepiped facing the paper.

The terminal device 20b aligns and synthesizes the first three-dimensional point cloud data and the second three-dimensional point cloud data to generate synthesized three-dimensional point cloud data as shown in FIG. 34 (step S14). As is clear from this figure, it is possible to obtain three-dimensional point cloud data with a larger amount of information (having both measurement data from the front and rear) than when the first three-dimensional point cloud data and the second three-dimensional point cloud data are used alone.

The composite 3D point cloud data obtained in this manner can be used for various purposes. In this embodiment, a 3D model of the feature is generated based on the composite 3D point cloud data.

The terminal device 20b generates the outer surface of the feature based on the composite 3D point cloud data (step S15). This process can be performed, for example, by a 3D convex hull algorithm (http://www.qhull.org/download/). Figure 35A shows the generated 3D model of the feature. It is possible to generate a 3D model that is more complete than a 3D model generated using only the first 3D point cloud data and a 3D model generated using only the second 3D point cloud data. The terminal device 20b displays this on the display 84 and records it on the hard disk 86 (step S15).

According to this embodiment, it is possible to easily search for multiple 3D point cloud data scattered on the Internet for the same feature, and to obtain composite 3D point cloud data by synthesizing the data. Also, if one attribute data is generated based on any one of the multiple 3D point cloud data for the same feature, the above search and synthesis can be performed. This is because the attribute data is independent of the 3D point cloud data, and the outer shape area data of the attribute data is defined in absolute coordinates.

3.4 Other
(1) In the above embodiment, two pieces of three-dimensional point cloud data are acquired to generate the composite three-dimensional point cloud data. However, three or more pieces of three-dimensional point cloud data may be acquired to generate the composite three-dimensional point cloud data.

(2) In the above embodiment, composite 3D point cloud data is obtained for one feature. However, composite 3D point cloud data may be obtained for multiple features. Also, composite 3D data may be obtained for all features within a specified range. Such examples are shown in Figures 36A, 36B, and 36C. Figure 36A shows the first 3D point cloud data, Figure 36B shows the second 3D point cloud data, and Figure 36C shows the composite 3D point cloud data.

(3) In the above embodiment, 3D point cloud data with different measurement directions is synthesized. However, 3D point cloud data with different measurement distances to features or 3D point cloud data with different measurement instruments (for example, measurements by MMS and UAV) may also be synthesized.

(4) In the above embodiment, a case where only one piece of attribute data is found for a location in step S21 has been described. However, there are cases where different measurement companies create 3D point cloud data for the same location and each uploads attribute data, or cases where the same company creates 3D point cloud data for the same location at different times and each uploads attribute data. In such cases, multiple pieces of attribute data are found for the specified location. In this case, the attribute data server device 60 transmits to the terminal device 20b that multiple pieces of attribute data exist. The terminal device 20b displays the creation date and time, creation device, etc. for the multiple pieces of attribute data on the display 84 and allows the operator to select one. Based on the selected attribute data, the outer shape area is transmitted (step S21).

(5) In the above embodiment, the terminal device 20b makes the 3D point cloud data request (step S13). However, the attribute data server device 60 may make this request and transmit the result to the terminal device 20b.

(6) In the above embodiment, the synthesis of the 3D point cloud data and the generation of the 3D model are performed in the terminal device 20b. However, either or both of these may be performed in the attribute data server device 60, and the results may be transmitted to the terminal device 20b.

(7) In the above embodiment, when the terminal device 20b acquires the three-dimensional point cloud data in step S14, it does not display from which direction the data was measured. However, to make this clearer, as shown in FIG. 35B, an arrow may be used to display the measurement direction and distance relative to the feature, based on the data attached to the acquired three-dimensional point cloud data. The base of the arrow indicates the measurement position, and the direction of the arrow indicates the measurement direction. In the case of FIG. 35B, it is clear that there is three-dimensional point cloud data measured from two locations.

Therefore, based on this screen display, it is clear that the next time a measurement is made, it is preferable to measure from a direction other than that shown by the arrow if the quality of the composite 3D point cloud data is to be improved. Also, if it is desired to know the changes over time of features, it is preferable to make measurements from the same position and direction as shown by the arrow.

(8) In the above embodiment, the first and second three-dimensional point cloud data are simply combined to generate the composite three-dimensional point cloud data. However, when the first three-dimensional point cloud data and the second three-dimensional point cloud data exist for the same location in the three-dimensional point cloud data, only the three-dimensional point cloud data with high measurement accuracy for that location may be used to generate the composite three-dimensional point cloud data.

(9) In the above embodiment, the first point cloud server device 40a and the second point cloud server device 40b are configured as separate server devices, but they may be configured as a single server device.

(10) In the above embodiment, the first point cloud server device 40a, the second point cloud server device 40b, and the attribute data server device 60 were constructed as separate server devices, but they may also be constructed as a single server device.

(11) The above-described embodiments and variations can be implemented in combination with other embodiments as long as this does not go against the essence of the invention.

4. Fourth embodiment
4.1 Overall Configuration Fig. 37 shows the overall configuration of a point cloud data management system according to a second embodiment of the present invention. A first three-dimensional point cloud data file at a first date and time based on absolute coordinates including features is recorded in the first point cloud server device 40a. A second three-dimensional point cloud data file at a second date and time based on absolute coordinates including features is recorded in the second point cloud server device 40b. Outline area data based on absolute coordinates that surrounds the outline of the features is recorded in the attribute data server device 60.

The terminal device 20 is capable of communicating with these server devices. The three-dimensional point cloud data acquisition means 22 of the terminal device 20 acquires the contour area data of the desired feature from the attribute data server device 60. The three-dimensional point cloud data acquisition means 22 accesses the first point cloud server device 40a based on the acquired contour area data of the feature, and acquires the three-dimensional point cloud data included in the area as the first three-dimensional point cloud data. Furthermore, the three-dimensional point cloud data acquisition means 22 accesses the first point cloud server device 40b based on the acquired contour area data of the feature, and acquires the three-dimensional point cloud data included in the area as the second three-dimensional point cloud data.

The time-dependent change acquisition means 25 calculates the change in shape of the feature between the first date and time and the second date and time based on the acquired first three-dimensional point cloud data at the first date and time and the acquired second three-dimensional point cloud data at the second date and time.

4.2 System Configuration The system configuration is the same as in the third embodiment.

4.3 Calculation of Changes over Time Next, a process for obtaining changes over time of features based on multiple published 3D point cloud data will be described.

Figures 38 and 39 show flowcharts for acquiring changes over time in features. In the figures, the terminal device is shown as a flowchart of the terminal program, the attribute data server device is shown as a flowchart of the attribute data server program, and the point cloud server device is shown as a flowchart of the point cloud server program.

First, the CPU 80 of the terminal device 20b (hereinafter sometimes abbreviated as terminal device 20b) accesses the attribute data server 60 and specifies the location where the desired feature is located (step S11). The location is specified by specifying a range using, for example, latitude and longitude. An address or place name may be entered (or searched), and this may be converted into a latitude and longitude range. Also, an area clicked or selected on the map may be converted into a latitude and longitude range.

In response to this, the CPU of the attribute server device 60 (hereinafter sometimes abbreviated as the attribute server device) searches the attribute data and extracts the contour area data within the specified location (step S21). The attribute server device 60 transmits the extracted contour area data (planar shape data and its position and height in absolute coordinates), feature name, etc. to the terminal device 20b.

The terminal device 20b displays the outline area and the feature name on the display 84 based on the received outline area data (step S12). An example of the display is shown in FIG. 32. The outline areas 102-112 of each feature are shown, and the feature name is shown in their vicinity.

The user operates the mouse 90 of the terminal device 20b to select the outline region of the desired feature. Here, the explanation will proceed assuming that the outline region 110 of the building has been selected. In response to this selection, the terminal device 20b specifies the outline region data of the outline region 110 of the building and transmits a request for three-dimensional point cloud data to each of the point cloud server devices 40a to 40n (step S13).

Each point cloud server device 40a-40n that receives this request determines whether it holds the three-dimensional point cloud data for the outer region 110, and if so, returns it. Here, the explanation will proceed assuming that two of the point cloud server devices 40a-40n, point cloud server devices 40a and 40b, hold the three-dimensional point cloud data for the outer region 110 of the building.

The CPU of the point cloud server device 40a (hereinafter sometimes abbreviated as point cloud server device 40a) returns the three-dimensional point cloud data of the exterior area 110 of the building to the terminal device 20b (step S31).

Similarly, the CPU of the point cloud server device 40b (hereinafter sometimes abbreviated as point cloud server device 40b) returns the three-dimensional point cloud data of the exterior area 110 of the building to the terminal device 20b (step S41).

The terminal device 20b receives three-dimensional point cloud data (first three-dimensional point cloud data) from the point cloud server device 40a and three-dimensional point cloud data (second three-dimensional point cloud data) from the point cloud server device 40b. Here, the first three-dimensional point cloud data and the second three-dimensional point cloud data are measurements of features from substantially the same direction. However, the measurement dates and times are different.

The terminal device 20b generates a three-dimensional model of the feature at the first date and time based on the first three-dimensional point cloud data (step S17). Also, the terminal device 20b generates a three-dimensional model of the feature at the second date and time based on the second three-dimensional point cloud data (step S17).

The terminal device 20b displays the generated three-dimensional model for the first date and time and the three-dimensional model for the second date and time on the display 84 so that they can be compared (step S18). For example, as shown in FIG. 40, a three-dimensional model 120 for the first date and time and a three-dimensional model 122 for the second date and time are displayed. Below that, the name of the feature and the date of measurement are displayed. Therefore, in the example of FIG. 40, it can be seen by comparing the changes in the external shape of the building on February 24, 2010 and on March 5, 2019.

To make comparison easier, the two may be displayed overlapping each other in different colors.

In addition, for example, if there is a situation where a feature does not exist at the first date and time but does exist at the second date and time (for example, a new road sign has been installed), then if there is measurement data for that location at the second date and time, then 3D point cloud data for the feature can be obtained. However, even if there is measurement data for that location at the first date and time, 3D point cloud data for the feature cannot be obtained. Therefore, by displaying a 3D model based on this, it becomes clear that there was no feature at the first date and time but that there is a feature at the second date and time.

For the purpose of the above-mentioned time-series comparison, it is preferable to use three-dimensional point cloud data in which the measurement direction relative to the feature is the same (or similar).

4.4 Other
(1) In the above embodiment, a 3D model is generated using 3D point cloud data at two dates and times, and then a comparison is performed. However, 3D point cloud data at three or more dates and times may be acquired, and a 3D model may be generated for each of the 3D models, and then a comparison may be performed.

(2) In the above embodiment, a comparison over time is performed for one feature. However, a comparison over time may be performed for multiple features. Also, a comparison over time may be performed for all features within a specified range.

(3) In the above embodiment, the terminal device 20b makes the 3D point cloud data request (step S13). However, the attribute data server device 60 may make this request and transmit the result to the terminal device 20b.

(6) In the above embodiment, the three-dimensional model is generated in the terminal device 20b. However, it may be generated in the attribute data server device 60, and the result may be transmitted to the terminal device 20b.

(7) In the above embodiment, when the terminal device 20b acquires the three-dimensional point cloud data in step S17, it does not display when each piece of three-dimensional point cloud data was measured. However, to make this clearer, the measurement date and time of each piece of three-dimensional point cloud data may be displayed based on the data attached to the acquired three-dimensional point cloud data, as shown in FIG. 41. When there is three-dimensional point cloud data with many dates and times, it is preferable to display it in this manner. The user can view this, select a certain number of dates that they wish to compare, and display the three-dimensional model.

(8) The above embodiment and modified examples can be implemented in combination with other embodiments as long as they are not contrary to the essence of the embodiment and modified examples. For example, when there are multiple 3D point cloud data from different directions at the same date and time (or the same month or year), composite 3D point cloud data may be generated to generate a 3D model, which may then be compared with the 3D model at another date and time.

(9) In the above embodiment, the first point cloud server device 40a and the second point cloud server device 40b are configured as separate server devices, but they may be configured as a single server device.

(10) In the above embodiment, the first point cloud server device 40a, the second point cloud server device 40b, and the attribute data server device 60 were constructed as separate server devices, but they may also be constructed as a single server device.

(11) In the above embodiment, the searched video data is played back on the terminal device. At this time, a map of the vicinity where the video was captured, three-dimensional point cloud data, etc. may be displayed together. Also, three-dimensional point cloud data as seen from an imaging position that changes over time may be displayed to match the playback of the video.

(12) In the above embodiment, three-dimensional models of features at two points in time are displayed for comparison. However, the normal vectors of the outlines of the three-dimensional models may also be displayed for both to make the angle changes of the surface easier to understand.

It is also possible to divide the area into small areas (25 cm cubic areas) and compare and display the number of points within each area. This allows the degree of roughness of the surface of features to be compared.

Furthermore, the reflection intensity of each point or area can be compared, allowing the smoothness of the surface to be compared.

You can also compare the volumes of three-dimensional models. This allows you to compare the growth of plants, the amount of leaves, etc.

(13) The above-described embodiments and variations can be implemented in combination with other embodiments as long as this does not go against the essence of the invention.

5. Fifth embodiment
5.1 Overall Configuration Fig. 42 shows a functional block diagram of a feature search system according to the fifth embodiment of the present invention. The recording unit 158 of the feature search server device 150 records attribute data such as feature names, outer area, and positions of the three-dimensional point cloud data, as well as video data.

The attribute data transmission means 150 of the feature search server device 150 reads out the feature name or outer area corresponding to the three-dimensional point cloud data recorded in the recording unit 158 and transmits it to the terminal device 40. The feature attribute data display means 182 of the terminal device 40 displays the feature name or outer area of the received feature on the display unit 188.

The operator operating the terminal device 40 uses the mouse 90 to view the feature name or outline area of the displayed feature and select the feature for which he or she wishes to search for video. The selection information transmission means 184 transmits to the feature search server device 150 which feature has been selected.

The feature position acquisition means 154 of the feature search server device 150 acquires the position of the selected feature. The video data transmission means 156 searches the video data in chronological order and extracts the portion with the imaging position information near the feature position. The extracted video data is transmitted to the terminal device 40. The video playback means 186 of the terminal device 40 plays the received video data on the display unit 188.

This makes it possible to easily search for and display locations where desired features are captured from a huge amount of video data.

5.2 System Configuration Figure 43 shows the system configuration of a point cloud data management system according to the third embodiment. Point cloud server devices 40a, 40b, ..., 40n are provided on the Internet. A feature search server device 62 is also provided. The point cloud server devices 40a, 40b, ..., 40n store three-dimensional point cloud data of measured features and video data (with image capture position information) captured during measurement. The feature search server device 62 stores attribute data as shown in Figure 11. The server devices 40a, 40b, ..., 40n, 60 are provided with terminal devices 20a, 20b, ..., 20n that are capable of communicating via the Internet.

The hardware configuration of the terminal device 20 is the same as that shown in Fig. 29. The point cloud server device 40 and the feature search server device 62 also have the same hardware configuration. However, in the point cloud server device 40, a point cloud server program is recorded instead of the terminal program 96. Also, in the feature search server device 62, a feature search server program is recorded instead of the terminal program 96.

5.3 Feature Search Processing A flowchart for feature search is shown in Figure 44. In the figure, what is shown as the terminal device is a flowchart of a terminal program, and what is shown as the feature search server device is a flowchart of a feature search server program.

First, the CPU 80 of the terminal device 20b (hereinafter sometimes abbreviated as terminal device 20b) accesses the feature search server device 62 and specifies the location where the desired feature is located (step S11). The location is specified by specifying a range using, for example, latitude and longitude. An address or place name may be entered (or searched), and this may be converted into a latitude and longitude range. Also, an area clicked or selected on the map may be converted into a latitude and longitude range.

In response to this, the CPU of the feature search server device 62 (hereinafter sometimes abbreviated as feature search server device 62) searches the attribute data and extracts the outline area data within the specified location (step S51). The feature search server device 62 transmits the extracted outline area data (planar shape data and its position and height in absolute coordinates), feature name, etc. to the terminal device 20b.

The terminal device 20b displays the outline area and the feature name on the display 84 based on the received outline area data (step S12). An example of the display is shown in FIG. 32. The outline areas 102-112 of each feature are shown, and the feature name is shown in their vicinity.

The user operates the mouse 90 of the terminal device 20b to select the outline region of the desired feature. Here, the explanation will proceed assuming that the outline region 106 of the sign has been selected. In response to this selection, the terminal device 20b specifies the outline region data of the sign's outline region 106 and transmits an instruction to the feature search server device 62 to search for the part of the video in which this sign is captured (step S19).

The feature search server device 62 reads the attribute data of the specified feature and obtains the location of the feature (position in absolute coordinates) (step S52). Next, the feature search server device 62 extracts the portion of the captured video data 162 in which the feature is captured based on this feature location (step S53).

Figure 45 shows the data structure of the captured video data 162. Time stamps TS1, TS2, ... indicating the time of capture are associated and recorded in association with the main video data 170. The position of the camera (or the car equipped with the camera) that captured the video is also recorded in absolute coordinates. Although not shown, the captured video data 162 is also associated and recorded with information on the imaging area of the video as a whole.

Figure 46 shows the concept of imaging area information. The imaging path R is shown by a solid line. It shows that imaging started from a starting point S and continued to an end point E. The imaging area AR is shown by a dashed line that circumscribes this imaging path R. The imaging area AR is specified, for example, by the absolute coordinates of the diagonal (imaging area information).

Figure 47 shows details of the video portion extraction process in step S53. Based on the position of the feature to be searched, the feature search server device 62 selects video data 162 whose measurement position is included in the imaging area (step S531). The video data 162 selected in this way may include a portion in which the feature is imaged. The feature search server device 62 searches each of the selected video data 162 to see if the imaged portion is present.

In step S533, the feature search server device 62 reads out the imaging positions recorded in the video data 162 in chronological order (step S533). In the example of Figure 45, imaging positions PS1, PS2, ... PSn would be read out.

The feature search server device 62 identifies the imaging position PSf that is closest to the location of the feature among the retrieved imaging positions PS1, PS2, ..., PSn (step S534). The feature search server device 62 determines whether the distance between the closest imaging position PSf and the feature position is equal to or less than a predetermined value (step S535). If it is equal to or less than the predetermined value, it means that the feature has been captured at that location. The feature search server device 62 extracts video data for a predetermined time before and after the identified imaging position PSf (step S536). If the distance between the imaging position PSf and the feature position exceeds the predetermined value, the video portion is not extracted.

The above process is performed for all videos selected in step S531 (steps S532, S537).

Once the video data portion has been extracted in this manner, the feature search server device 62 transmits it to the terminal device 20b (step S54).

In response to this, the terminal device 20b displays the extracted portion of the video data on the display 84 (step S20). Note that if there is more than one video data, it is preferable to display a list of the video data together with the shooting date and time before playback, and allow the user to select one to play.

The user can view this video and understand the condition of the feature that was the subject of the search as an image. For example, this can be used when wanting to know the condition of the feature that can be identified using the screen display in Figure 32 (checking looseness detection marks (e.g., match marks) on bolts and nuts, checking the deterioration of the appearance, etc.).

In addition, the above detection marks and the appearance deterioration status may be checked by the terminal device using image processing or AI.

5.4 Other
(1) In the above embodiment, the feature positions in the attribute data and the capture positions in the video data are expressed in absolute coordinates. Therefore, even if attribute data corresponding to each piece of video data is not prepared, one piece of attribute data is enough to search for features in multiple pieces of video data.

If attribute data is provided for each video data, the feature position and image capture position do not need to be absolute coordinates as long as they are consistent between the two data. Also, if there is attribute data for the video data, it is possible to perform searches based on the date and time when the 3D point cloud data of the feature was created and the date and time when it was photographed, rather than searches based on the feature position and image capture position.

(2) In the above embodiment, the feature to be searched for is specified by the outline area and the feature name as shown in FIG. 32. However, it is also possible to display only the feature name without displaying the outline area and to allow the user to select it.

(3) In the above embodiment, the portion of the video in which the feature is captured is extracted and played back. However, in addition to this, the feature may be marked in the video and played back.

The feature search server device 62 can achieve this by carrying out the following process. For example, suppose there is an outline area of a feature as shown in FIG. 48, and a sign 106 is the search target. In the space in which this outline area is located, a trajectory 200 is drawn when a video is captured. Since time is added to this trajectory 200, it is possible to know the image capture time at each point. At each image capture time t1, t2, ..., it is determined in which direction and at what distance the target feature 106 exists.

Based on this direction and distance, it calculates how the feature 106 is captured at the corresponding time in the captured video. The appearance of the outer region of the feature 106 generated in this way at each time is superimposed on the extracted video and transmitted to the terminal device 20b.

When this is played back on the terminal device 20b, as shown in FIG. 49, a frame of the outline area is displayed to surround the desired feature in the image. Therefore, the user can easily find the desired feature in the video.

(4) In the above embodiment, the point cloud server device 40 and the search server device 62 are constructed as separate server devices, but they may also be constructed as a single server device.

(5) In the above embodiment, a case where videos are searched for is described. However, still images with the shooting location and shooting time can also be searched for using the same process.

(6) The above-described embodiments and variations can be implemented in combination with other embodiments as long as this does not go against the essence of the invention.

Claims (6)

A feature management system including a server device and a mobile terminal device capable of communicating with the server device,
The server device includes:
a recording unit that records three-dimensional point cloud data including features, as well as past imaging positions and imaging directions;
a past two-dimensional projection image generating means for generating a past two-dimensional projection image by projecting the three-dimensional point cloud data of the feature or the three-dimensional model thereof onto a plane based on the past imaging positions and imaging directions, and transmitting the generated past two-dimensional projection image to a terminal device;
The terminal device
A feature management system comprising: a display means for displaying, on a display unit, a past two-dimensional projection image received from a server device superimposed as a transparent image on a viewfinder image of a camera for capturing images of the feature. a recording unit that records three-dimensional point cloud data including features, as well as past imaging positions and imaging directions;
a past two-dimensional projection image generating means for generating a past two-dimensional projection image by projecting the three-dimensional point cloud data of the feature or the three-dimensional model thereof onto a plane based on the past imaging positions and imaging directions, and transmitting the past two-dimensional projection image to a mobile terminal device;
A server device comprising: 3. The server device according to claim 2,
The server device has a plurality of devices that can communicate with each other,
The plurality of devices are characterized in that they include a point cloud server device that records the three-dimensional point cloud data, and an attribute server device that has the past two-dimensional projection image generation means and records the past imaging positions and imaging directions. A server program for implementing a server device by a computer, the server program comprising:
A server program for functioning as a past two-dimensional projection image generation means that projects the three-dimensional point cloud data of the feature or its three-dimensional model onto a plane based on the three-dimensional point cloud data including the feature recorded in the recording unit and the past imaging positions and imaging directions, thereby generating a past two-dimensional projection image and transmitting it to a terminal device. The server program according to claim 4,
The server device has a plurality of devices that can communicate with each other,
A server program characterized in that the plurality of devices include a point cloud server device that records the three-dimensional point cloud data, and an attribute server device that has the past two-dimensional projection image generation means and records the past imaging positions and imaging directions. Record three-dimensional point cloud data including features, as well as past imaging positions and imaging directions,
generating a past two-dimensional projection image by projecting the three-dimensional point cloud data of the feature or the three-dimensional model thereof onto a plane based on the past imaging positions and imaging directions;
A feature management method characterized in that a past two-dimensional projected image is superimposed and displayed on a camera viewfinder image.

Images