WO2020189909A2

WO2020189909A2 - System and method for implementing 3d-vr multi-sensor system-based road facility management solution

Info

Publication number: WO2020189909A2
Application number: PCT/KR2020/002651
Authority: WO
Inventors: 박일석; 홍승환; 송승관
Original assignee: 주식회사 스트리스
Priority date: 2019-03-15
Filing date: 2020-02-25
Publication date: 2020-09-24
Also published as: WO2020189909A3

Abstract

Provided are a system and method for implementing a 3D-VR multi-sensor system-based road facility management solution. The system comprises: a sensor data collecting unit for collecting observation data and temporal information of a road facility from an image sensor, a navigation sensor and a 3D measurement sensor for obtaining 3D geographic data, respectively; a data convergence unit for performing data convergence for geometric models on the basis of geometric structure information comprising an inner geometry and an outer geometry defined for each sensor; an image data processing unit for generating image data for spatial information by registering, in pixel units of image data, 3D spatial information generated from point cloud data-related information comprising at least 3D location data with reference to the index, geometric structure information and temporal information; an object detection and recognition unit for detecting and recognizing object information related to the road facility on the basis of the image data for spatial information; and a database unit for linking and storing the image data for spatial information and the object information.

Description

System and method for implementing road facility management solution based on 3D-VR multi-sensor system

The present invention relates to a system and method for implementing a road facility management solution based on a 3D-VR multi-sensor system, and in more detail, the status and maintenance of road facilities such as roads for automobiles, bicycles, on foot, and roads around rivers. In order to easily and accurately check the part, a 3D surveying sensor such as a laser sensor and a navigation sensor are combined around the image sensor to register accurate 3D spatial information in each pixel of 2D image data to provide an image pixel-based service. It relates to a system that does.

Since road facilities such as automobile roads, bicycle roads, footpaths, and roads around rivers are managed by region, a lot of manpower and organization are required, but there is a limit to the management of road facilities due to insufficient financial resources. In order to overcome this, there is a demand for a technology that minimizes the cost and manpower required for on-site survey on whether there is a repair part.

With this technology, photographing of road facilities using an image sensor mounted on a ground or aerial mobile platform is being utilized.

Specifically, in the case of providing the current status of road facilities in a panoramic image by mounting multiple camera sensors, if an overpass and a facility spaced apart from the ground overlap with a facility located on the ground, the same two-dimensional location data in the image is superimposed facility. It is not possible to check various information of the house, and since the image only holds the location data of 2D coordinates, various analysis due to the height difference between the two facilities is impossible. In addition, in the case of using a wide-area image sensor having a large field of view so that the image data includes more facilities, the image data is generated as a panoramic image that can be implemented in 3D-VR, such as a head mounted display or a head-up display. Is distorted compared to the flat image (refer to FIG. 6(b)), so it is not easy to extract object information.

In addition, when the mobile platform on which the image sensor is mounted is a drone, the utility is degraded due to the limitation of flight altitude in the city center where high-rise buildings and road facilities are concentrated. In addition, in the case of viewing by a drone, there is an obstructed area where the current status of the ground facilities cannot be photographed due to bridges, buildings, and facilities spaced apart from the ground, so that all road facilities cannot be photographed.

Meanwhile, a mobile mapping system equipped with heterogeneous sensors is sometimes used as a technology to check the status and maintenance of road facilities.

The mobile mapping system is an integrated sensor system that combines navigation sensors, image sensors, and laser sensors of each unique coordinate system, and geo-referencing and calibration that integrates each coordinate system into one data. Through the (calibration) operation, point cloud data containing 3D spatial information is created.

Conventionally, digital maps in the form of vectors or models, precision road maps, and 3D models are generated through drawing and modeling processes based on 3D point cloud data.

This is expressed by recording 3D shape information through a laser sensor capable of recording 3D coordinates with an accuracy of mm level, and georeferencing the laser sensor data through a combination with a navigation sensor to give spatial coordinates.

Here, the image information is used for coloring each point cloud data acquired from the laser sensor.

However, in the case of an image, since the scale value is different for each pixel, accurate 3D spatial information such as 3D position data and geometric information such as internal and external geometry cannot be obtained from the image. The use of image data in the mobile mapping system is used only at the level of inverse calculation of values such as length, location, and area of an object.

In the case of an image, in order to register 3D spatial information in an image pixel, it is necessary to know the exact position and posture value of the image sensor, but a separation occurs due to a mismatch between the center of the navigation sensor and the image sensor.

As a solution to this, there is a limitation in minimizing the separation distance between the navigation sensor, the image sensor, and the laser sensor, or matching or overlapping the viewing angle of the image sensor and the laser sensor.

On the other hand, artificial intelligence and machine learning algorithms for recognizing objects are often developed to be applied to flat image data, so there are practically no algorithms for panoramic images generated as distorted images.

Therefore, in order to effectively utilize the mobile mapping system, the development of a new technology for applying most of the image-based artificial intelligence and machine learning algorithms to the mobile mapping system is required.

The technical task to be achieved by the present invention is to provide accurate information on each pixel of 2D image data by fusing a 3D survey sensor such as a laser sensor and a navigation sensor around the image sensor so that the current status and maintenance of road facilities can be easily and accurately checked. It is to provide a system that provides a service based on an image pixel by registering 3D spatial information.

The object of the present invention is not limited to the object mentioned above, and other objects that are not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention for achieving the above technical problem, a system for implementing a road facility management solution based on a 3D-VR multi-sensor system includes an image sensor, a navigation sensor, and a 3D survey sensor that acquires 3D geographic data. A sensor data collection unit that collects observation data and time information for road facilities, an internal geometry calculated based on a geometric parameter defined for each sensor, and angles according to the position and attitude of the platform on which at least one of the sensors is mounted. A data fusion unit that performs data fusion for a geometric model based on geometry information consisting of external geometry defining a geometric relationship between sensors, and a pixel of image data from observation data of the image sensor and the 3D survey sensor. Generated from point cloud data-related information including at least the three-dimensional position data by referring to the index given to define a correspondence relationship between the three-dimensional position data of point cloud data-related information from observation data, the geometric structure information, and the time information An image data processing unit that generates image data for spatial information by registering the 3D spatial information in pixel units of the image data, and detecting and recognizing object information related to the road facility based on the image data for spatial information. An object detection and recognition unit, and a database unit for storing the image data for spatial information and the object information in association with each other.

In another embodiment, the point group data-related information includes 3D position data, point group color information, point group class information estimated from the object extracted from the 3D position data, and the 3D survey sensor. It may include at least one of laser intensity information in the case of.

In another embodiment, the object detection and recognition unit detects and recognizes a maintenance part of the road facility along with the road facility based on the image data for spatial information through machine learning, and the object information is the road facility And attribute data including meta data related to the type, name, and detailed information of the object, shape, shape, and texture, in relation to the repair part.

Here, the database unit may store 3D map data including at least repair section location data based on the spatial information image data linked with the object information.

In another embodiment, further comprising a manipulation/display unit capable of editing predetermined data while providing the spatial information image data linked to the object information to a user in a screen form, wherein the manipulation/display unit is displayed by the user on the screen. When at least a part of a specific road facility is designated in an image, control to display at least the name of the road facility among 3D location data and the object information of the specific road facility, or the 3D location data and the object information Interest for controlling the user to edit at least one of them, and controlling the manipulation/display unit to display the region of interest in an enlarged screen form when the user selects a part of the image as the region of interest It may further include a local screen generator.

Here, the image data processing unit, the object detection and recognition unit, and the database unit may be implemented in a high-load processing module, and the display/manipulation unit can be implemented in a low-load processing module having a lower computational processing capability than the high-load processing module. have.

In another embodiment, by obtaining external data including object information including a plurality of attribute data, extracting object information including at least metadata related to location data and detailed information of the road facility from the external data, , When at least one of a plurality of attribute data of object information of the spatial information image data recorded at the same location is different from the object information of the external data, the spatial information image data is the object information of the external data It may further include an update unit to update to.

Here, when the image sensor is configured as a wide area image sensor having a wider field of view than a flat image sensor generating a flat image, image data from observation data of the wide area image sensor is generated as a panoramic image, and Further comprising a distortion removal unit for removing distortion, the distortion removal unit index-based distortion removal unit for performing index-based image distortion removal when the image data is input, and a geometric model-based image distortion removal when the image data is input And a geometric model-based distortion removing unit for performing, wherein the distortion removing unit includes the index-based distortion removing unit and the geometric model-based distortion according to at least one of a type of the wide area image sensor, a good degree of distortion removal, and a user's selection. Image distortion can be removed by any one of the removal units.

In addition, the index-based distortion removal unit may include a pixel index assigning unit for assigning an index to each pixel to define a correspondence relationship between the distorted image data from the image data and the reference plane image data, and the distortion image to which the index is assigned. A section decomposition unit that decomposes data into preset sections, and a planar image that generates flat image data by removing distortion of the distorted image data based on a distortion correction model of the decomposed section by referring to the reference plane image data It may include a generating unit.

In addition, the geometric model-based distortion removal unit defines a 3D shape restoration unit for restoring a virtual 3D shape from the camera geometry-based distortion image data based on the distortion image data from the image data, and a virtual camera geometry. And a planar image generator configured to generate planar image data by projecting it onto a virtual planar camera through the geometrical equation.

In another embodiment, facility information obtained by obtaining and processing facility information corresponding to the road facility from the outside based on the 3D location data belonging to the 3D spatial information and 2D location data included in the image data Further comprising an acquisition unit, wherein the facility information acquisition unit obtains a time-series aerial orthogonal image including the facility information related to the road facility from the outside, and extracts geometric line form information from the aerial orthogonal image. , Based on the road linear information, facility structure information and accessories are time-sequentially processed into a document or data sheet, and the basic road information is generated, and the facility information acquisition unit 2 accumulated in the form of an image in relation to the road facility. Dimensional drawing data and 3D models are received from the outside, and drawing information is generated by digitizing the drawing data and the 3D model by image-based object recognition and 3D model recognition, and the facility information acquisition unit generates drawing information in relation to the road facility. By obtaining the status/maintenance document and data sheet accumulated in the form of an image, and digitizing the document and the data sheet by image-based text recognition, road status/maintenance information is generated, and the basic road information, the drawing information, and the Road status/maintenance information may be stored in the database unit in association with the road facility.

According to another aspect of the present invention for achieving the above technical problem, a method for implementing a road facility management solution based on a 3D-VR multi-sensor system is provided from an image sensor, a navigation sensor, and a 3D survey sensor that acquires 3D geographic data. Collecting observation data and time information for road facilities, internal geometry calculated based on geometric parameters defined for each sensor, and geometry between each sensor according to the position and attitude of the platform on which at least one of the sensors is mounted Performing data fusion for a geometric model based on geometry information consisting of external geometry defining a relationship, and a point cloud from pixels of image data from observation data of the image sensor and observation data of the 3D survey sensor 3D spatial information generated from point cloud data related information including at least the 3D position data by referring to the index given to define a correspondence relationship between 3D position data of data related information, the geometry information and the time information Generating spatial information image data by registering the image data in pixel units, detecting object information related to the road facility based on the spatial information image data, and the spatial information image data And storing the object information in association.

Details of other embodiments are included in the detailed description and drawings.

According to the present invention, a three-dimensional survey sensor such as a laser sensor and a navigation sensor are combined with a three-dimensional survey sensor such as a laser sensor centered on an image sensor so that the current status and maintenance of road facilities can be easily and accurately identified, thereby providing an accurate three-dimensional space for each pixel of the two-dimensional image data. By registering information, an image pixel-based service can be provided.

In addition, by defining the geometry of the system around the image sensor, it improves accuracy and shortens processing time.

According to the present invention, image-based machine learning and artificial intelligence algorithms can be applied to the system, thereby reducing drawing and modeling work time and improving accuracy.

In addition, when the viewing angle between the three-dimensional survey sensor and the image sensor such as a laser used LiDAR (Light Detection and Ranging; LiDAR) does not match, the data of the navigation sensor and the time information according to the observation of each sensor are used. 3D spatial information can be given to image pixels.

In addition, each sensor and data are indexed and linked together to shorten data processing time and reduce data capacity.

1 is a block diagram of a system for implementing a road facility management solution based on a 3D-VR multi-sensor system according to an embodiment of the present invention.

2 is a diagram illustrating a ground mobile platform and sensors mounted thereon.

3 is a configuration diagram of a distortion removing unit.

4 is a block diagram of an object detection and recognition unit.

5 is a flowchart illustrating a method of implementing a road facility management solution based on a 3D-VR multi-sensor system according to an embodiment of the present invention.

6 is a diagram illustrating the concept of a geometric model for generating geometric structure information.

7 is a diagram illustrating a distorted panoramic image as image data captured from a wide area image sensor.

8 is a flowchart illustrating a process of removing image distortion by an index-based distortion removing unit.

9 is a flowchart illustrating a process of removing distortion of an image by a geometric model-based distortion removing unit.

10 and 11 are diagrams schematically illustrating a process of removing image distortion by an index-based distortion removing unit.

12 is a diagram schematically illustrating a process of removing image distortion in a geometric model-based distortion removing unit.

13 is a diagram illustrating image data from which distortion is removed by a distortion removal unit.

14 is a diagram illustrating image data for spatial information generated by registering 3D spatial information in pixel units of image data.

15 is a diagram illustrating a screen in which a part of the location data and object information of the road facility is output to the manipulation/display unit when a user designates a specific road facility in an image provided to the manipulation/display unit.

16 illustrates that the object detection and recognition unit detects and recognizes a maintenance part of a road facility based on image data for spatial information by machine learning, reads it as object information, and stores 3D maintenance section position data in the database unit. It is a drawing.

17 is a diagram illustrating information provided when image data for spatial information including object information is provided to a manipulation/display unit.

18 is a flowchart illustrating a process of updating image data for spatial information when external data including object information different from the image data for spatial information is acquired by an updater according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings and the following description. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content may be thorough and complete, and the spirit of the present invention may be sufficiently conveyed to those skilled in the art. The same reference numbers throughout the specification denote the same elements. Meanwhile, terms used in the present specification are for explaining embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, actions and/or elements in which the recited component, step, action and/or element is Or does not exclude additions.

In addition, "unit" to "module" generally refer to components such as software (computer program) and hardware that can be logically separated. Therefore, the module in this embodiment refers not only to the module in the computer program, but also to the module in the hardware configuration. Therefore, the present embodiment is a computer program for making them function as modules (a program for executing each step in a computer, a program for making a computer function as each means, a program for realizing each function in a computer) ), also explains the system and method. However, for convenience of explanation, phrases equivalent to "save" and "save" are used, but these words are stored in a memory device or stored in a memory device when the embodiment is a computer program. It means to control like that. In addition, the "sub" to the module may correspond to a function one-to-one, but in mounting, one module may be configured as one program, multiple modules may be configured as one program, and conversely, one module may be configured as multiple programs. do. Further, a plurality of modules may be executed by one computer, or one module may be executed by a plurality of computers by a computer in a distributed or parallel environment. In addition, one module may contain other modules. In addition, hereinafter, "connection" is adopted in the case of logical connection (data exchange, instruction, reference relationship between data, etc.) in addition to physical connection. "Pre-determined" means that it is determined before the target processing, and if it is before the target processing, not only before the processing according to the present embodiment is started, but also after the processing according to the present embodiment is started, It includes the meaning of what is determined according to the situation and state of the time, or the situation and state up to that time.

In addition, a system or device is configured by connecting a plurality of computers, hardware, devices, etc. by communication means such as a network (including one-to-one communication connection), and a single computer, hardware, device, etc. This includes cases where it is realized. The terms "device" and "system" are used as terms of mutual agreement. Of course, the "system" does not include anything but an artificial decision, a social "mechanism" (social system).

In addition, when processing by each unit or module or a plurality of processing within each unit or module, target information is read and input from the storage device for each processing, and after the processing is performed, the processing result is displayed. It is writing to the memory device. Accordingly, descriptions of read input from the storage device before processing and write to the storage device after processing may be omitted in some cases. Further, the storage device herein may include a hard disk, a random access memory (RAM), an external storage medium, a storage device via a communication line, a register in a CPU (Central Processing Unit), and the like.

Hereinafter, a system for implementing a road facility management solution based on a 3D-VR multi-sensor system according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 4.

1 is a block diagram of a system for implementing a road facility management solution based on a 3D-VR multi-sensor system according to an embodiment of the present invention. 2 is a diagram illustrating a ground mobile platform and sensors mounted thereon, FIG. 3 is a configuration diagram of a distortion removing unit, and FIG. 4 is a configuration diagram of an object detection and recognition unit.

System 100), which implements a road facility management solution based on a 3D-VR multi-sensor system, is equipped with heterogeneous sensors on platforms such as ground and/or aerial vehicles, and processes and analyzes data from the sensors to determine the status of road facilities and It may be a mobile mapping system that recognizes and analyzes the maintenance part. Road facilities may be facilities such as automobile roads, bicycle paths, walking paths, and roads around rivers, but do not exclude various types of facilities installed on the ground.

As shown in FIG. 1, the system 100 includes an image sensor 102, a 3D survey sensor 104, a sensor data collection unit including a navigation sensor 106, a data fusion unit 108, and a distortion removal unit 110. ), image data processing unit 112, object detection and recognition unit 114, database unit 116, operation/display unit 118, region of interest screen generation unit 120, update unit 122 and facility information acquisition unit (123) may be included.

The image sensor 102 is mounted on a moving ground or aerial mobile platform to capture an image of surrounding objects, such as terrain and features, and acquire observation data for images and time information for images at the time the observation data was acquired. It is a sensor, and may be a wide area image sensor having a wider field of view than a planar image sensor that generates a flat image, a surveying or non-surveying camera, and a stereo camera, but is not limited thereto. The wide-area image sensor generates a panoramic image that can be implemented in a 3D-VR, such as a head mounted display or a head-up display. For example, an omnidirectional image by a camera that generates a fisheye lens image with a fisheye lens or a multi-camera using multiple cameras It may be a multi-camera system that outputs.

The 3D survey sensor 104 is mounted on a platform to acquire 3D geographic data related to objects around it, such as terrain and feature related data, and is used for 3D surveying observation data and 3D surveying at the time the observation data was acquired. As a sensor that acquires time information, it is an active remote sensing sensor. For example, the 3D survey sensor 104 may be a laser or ultrasonic sensor, and in the case of a laser sensor, it may be a lidar sensor. Such a lidar sensor scans a laser on an object to obtain data and detects the parallax and energy change of the electromagnetic wave reflected from the object and returned, and calculates the distance to the object and the reflection intensity.

The navigation sensor 106 is a sensor that detects navigation information such as positioning information, position of the platform, attitude, and speed, and acquires observation data for navigation and time information at the time the observation data was acquired, and uses a satellite navigation device (GPS). It can be composed of a position acquisition device that acquires the moving position of the platform through an inertial measurement unit (IMU), an attitude acquisition device that acquires the attitude of the vehicle through an inertial navigation system (INS). .

Each sensor may be mounted on the same platform, but may be distributed and mounted on a ground mobile platform and an aerial platform such as satellite, aerial, and drone. The ground mobile platform 10 may be a vehicle, a bicycle, a two-wheeled vehicle, or equipment separately configured for walking, and the three-dimensional survey sensor 104 and navigation sensor 106 composed of an image sensor 102 and a lidar sensor are shown in FIG. 2 It may be mounted on the ground mobile platform 10 as described above.

The sensor data collection unit provided with each sensor 102 to 106, along with observation data and time information acquired from each sensor 102 to 106, and unique data related to device-specific properties for each sensor 102 to 106, each sensor (102-106) Alternatively, acquired environment information including weather, wind speed, temperature at which the sensor 102-106 or the platform is located, and humidity, which appear when observation data from the platform is acquired, may be collected.

The unique data of the image sensor 102 includes at least one of sensor illumination, ISO, shutter speed, shooting date/time, time synchronization information, sensor model, lens model, sensor serial number, image file name, and file storage location. can do.

The peculiar data of the 3D survey sensor 104 include LiDAR information, for example, the sensor's photographing date and time, sensor model, sensor serial number, laser wavelength, laser intensity, laser transmission/reception time, laser observation angle/distance, laser pulse. , May include at least one of an electronic time delay, a standby delay, and a sensor temperature.

The unique data of the navigation sensor 106 includes sensor model information of GNSS/IMU/INS, GNSS reception information, GNSS satellite information, GNSS signal information, GNSS navigation information, ionospheric/convection layer signal delay information of GNSS signals, DOP information, and Earth. Behavior information, dual/multi GNSS equipment information, GNSS base station information, wheel sensor information, gyro sensor scale/bias information, accelerometer scale/bias information, position/position/speed/acceleration/angular velocity/angular acceleration information and predicted error amount, navigation It may include at least one of all or part of the information filtering model and erasure error information, a photographing date/time, and time synchronization information.

The data fusion unit 108 includes the observation data, the internal geometry calculated based on the geometric parameters defined for each of the sensors 102 to 106, and the position of the platform 10 on which at least one of the sensors 102 to 106 is mounted. Data fusion for a geometric model is performed based on geometric structure information composed of an external geometry that defines a geometric relationship between each sensor 102 to 106 according to a posture.

The data fusion unit 108 generates point cloud data related information related to the 3D survey sensor 104 based on observation data, geometric structure information, and time information, and is generated from the data of the image sensor 102 to perform geocoding. ) Image data including image information may be generated. In the present embodiment, point cloud data-related information and geocoding image information are generated by the data fusion unit 108 as an example, but may be generated by the image data processing unit 112.

Specifically, the data fusion unit 108 calculates an internal geometry based on a geometric parameter defined for each of the sensors 102 to 106, and in this case, it may be calculated through a predetermined equation or modeling.

The internal geometry is an intrinsic value of the sensor itself, and is an error in observed data for each sensor 102 to 106 due to a parameter maintained regardless of whether the platform or the like is moved.

The image sensor geometric parameter may be at least one of a focal length, a main point position, a lens distortion parameter, and a sensor format size.

The 3D survey sensor geometric parameter may be at least one of an incidence angle of each laser, a distance scale, a distance direction offset, and an axial offset, for example, a lidar sensor.

In addition, the geometric parameter of the navigation sensor may be at least one of a scale in an axial direction and an offset in an axial direction.

The external geometry is calculated based on parameter values that change each time observation data of each sensor 102 to 106 is acquired due to the position and attitude of the platform being moved, that is, the position and attitude of each sensor 102 to 106 It is an error of the observed data, and in this case, it can be calculated through a predetermined equation or modeling.

The point group data related information is at least one of 3D position data, point group color information, point group class information whose type is estimated from the object extracted from 3D position data, and laser intensity information when the 3D survey sensor is a laser sensor. It may include.

The image data may be an RGB image or a black and white image data, and the geocoding image information included in the image data is a kind from the object extracted from the coordinate data for geocoding, color information for geocoding, and coordinate data for geocoding for each pixel. May include at least one of the estimated geocoding class information.

On the other hand, when the image sensor 102 is composed of a wide area image sensor having a wider field of view than a flat image sensor generating a flat image, the distortion removing unit 110 is the image data from the observation data of the wide area image sensor 102 Is generated as a panoramic image, and distortion of the panoramic image (refer to FIG. 6(b)) may be removed.

Specifically, the distortion removal unit 110 includes an index-based distortion removal unit 110 that performs index-based image distortion removal when image data is input, and a geometric model-based image distortion removal unit when image data is input. A model-based distortion removal unit 110 may be provided.

The distortion removal unit 110 is based on at least one of a type of a wide area image sensor, a good degree of distortion removal, a user's selection, and other options. Image distortion may be removed by either the index-based distortion removal unit 110 or the geometric model-based distortion removal unit 110.

For example, when the wide area image sensor is a camera equipped with a fisheye lens, image data may be input to the index-based distortion removal unit 110 based on the type of the wide area image sensor, and image distortion may be removed. .

In addition, in the case of a multi-camera system in which the wide-area image sensor outputs an omnidirectional image, the image data is converted to the index-based distortion removal unit 110 and the geometric model-based distortion removal unit 110 according to the level of goodness of distortion removal and the user's selection. ) Can be input to remove image distortion.

The index-based distortion removal unit 110 includes a pixel index provision unit 128, a section decomposition unit 130, and a planar image generation unit 122 to remove image distortion when image data for a fisheye lens is input. can do. In addition, the index-based distortion removal unit 110 may further include a section re-decomposition unit 134 in addition to the above-described member to remove image distortion when omnidirectional image data is input.

As shown in FIG. 10, the pixel index assigning unit 128 may assign an index to each pixel to define a correspondence relationship between the distorted image data from the image data and the reference plane image data.

The section decomposition unit 130 may decompose the distorted image data to which the index is assigned into a preset section.

The planar image generator 122 may generate planar image data by removing distortion of the distorted image data based on a distortion correction model of the disassembled section by referring to the planar image data for reference. The distortion correction model may be modeling based on an equation or index of the decomposed section.

The section re-decomposition unit 134 is a member used in the case of omnidirectional image data, and may re-decompose the omnidirectional image decomposed by the section decomposition unit 130 into a preset section, as in the third step of FIG. 11. . In this case, the planar image generator 122 may remove the distortion based on a distortion correction model of the section re-decomposed by the section re-decomposition unit 134, for example, based on an equation or an index.

The geometric model-based distortion removal unit 110 includes a 3D shape restoration unit 136 for restoring a virtual 3D shape from the distortion image data based on a camera geometry based on the distortion image data from the image data and a virtual camera geometry. It may include a planar image generator 138 for generating planar image data by defining and projecting it to a virtual planar camera through the geometrical equation.

The distortion removal unit 110 for removing distortion of an image for a fisheye lens and an omnidirectional image according to the present invention relates to lens distortion removal for creating an image data set for application to machine learning.

The distortion removal according to the physical lens structure is performed based on the geometric model or index of each sensor, and the projection distortions that change according to the image projection angle are converted to have geometric characteristics similar to those of the plane image, and are applied to the plane image. Make it possible to use the model as it is.

The geometric model-based image distortion removal method is performed by restoring a virtual 3D object from an image with distortion and reprojecting it onto a virtual plane camera. This is a method of solving the interrelationship of the plane images of by giving an index to each pixel.

The index-based method is more efficient in processing time than the geometric model-based method when processing multiple images.

In this embodiment, the system 100 includes the distortion removal unit 110 as an example, but in another embodiment, the system 100 does not include the distortion removal unit 110 and image data is processed into image data. A panoramic image, which is directly input to the unit 112 and without distortion, may be provided to the manipulation/display unit 118.

Meanwhile, the image data processing unit 112 stores at least the 3D position data with reference to the index, geometry information and time information given to define a correspondence relationship between the pixel of the image data and the 3D position data of the point cloud data related information. Image data for spatial information is generated by registering 3D spatial information generated from the included point cloud data-related information in pixel units of the image data.

The 3D spatial information may include not only 3D location data but also the above-described various data constituting point cloud data related information in order to supplement image data including only 2D location data in the coordinate data for geocoding. Image data for spatial information may be configured by combining image information for geocoding and 3D spatial information for each pixel.

The index may be assigned to define a correspondence relationship between the pixel of the image data and the 3D position data of the point cloud data related information by absolute coordinates of the point cloud data related information and the image data with reference to the navigation data. In another embodiment, the index refers to the coordinate system (S) of the 3D survey sensor 104 and the coordinate system (C) for the image sensor, the class of the data, the object shape, etc., and matches the point cloud data-related information and image data, It may be given to set a virtual absolute coordinate system between them to define a correspondence relationship between pixels and 3D position data according to the absolute coordinate system.

According to this, an index is assigned to each sensor and data and linked to each other to shorten data processing time and reduce data capacity.

When the viewing angle of the image sensor 102 and the 3D survey sensor 104 does not match, or the viewing angle of the 3D survey sensor 104 does not sufficiently cover the view angle range of the image sensor 102, the image data processing unit 112 may register 3D spatial information in pixel units of image data using the navigation data and time information of each of the sensors 102 to 106.

A model for imparting 3D spatial information to pixels of image data may be defined as in Equation 1, and some equations and variables may be modified or omitted during calculation.

(Equation 1)

Variables included in each image sensor 102, 3D survey sensor 104, navigation sensor 106, and system model are determined by map values or a calibration process to determine the accuracy and precision of 3D spatial information and data fusion. .

Here, when image data is displayed in the form of an RGB image or a black-and-white image, 3D spatial information may be given to each pixel or time, index, and navigation data may be assigned to each pixel. To this end, the image data processing unit 112 may link the entire geocoding image information, 3D spatial information, time information, index, and navigation data for each pixel.

Accordingly, when editing image data, it is possible to edit based on an RGB image or a black and white image, and perform 3D drawing or modeling using 3D spatial information registered without error for each pixel accurately corresponding to the 3D position data. I can.

Meanwhile, the object detection and recognition unit 114 may detect and recognize object information related to road facilities and maintenance parts of facilities based on image data for spatial information through machine learning. The object detection and recognition unit 114 detects and recognizes a maintenance part of a facility by referring to a facility recognition unit 140 that detects and recognizes object information of a road facility and object information of a road facility and image data for spatial information of the facility. A partial recognition unit 142 may be provided.

The object information may include attribute data including a type, a name, and metadata related to detailed information, a shape, a shape, and a texture in relation to a road facility and a maintenance part.

The facility recognition unit 140 and the maintenance part recognition unit 142 detect candidate group data related to the object identified from the spatial information image data using a machine learning technique, and sequentially apply an additional machine learning model to the detected candidate group data. The information of the object can be recognized.

The object detection and recognition unit 114 may perform preprocessing to improve the accuracy of object detection and recognition in image data for spatial information before the machine learning technique. Pre-processing of image sensor data for spatial information includes, for example, image brightness adjustment, image color adjustment, image blurring, image sharpness, image texture analysis, and observation geometric characteristics. It may be performed by at least one of movement, rotation, and/or scale transformation of image sensor data due to the difference (converting one or more of Rigid-body, Affine, and Similartiy as 2D transformed data).

The database unit 116 links and stores image data for spatial information, road facilities, and object information of a maintenance part. In addition, the database unit 116 may store 3D map data including at least maintenance section position data based on image data for spatial information linked to object information, as shown in the left drawing of FIG. 16.

The user may access the database unit 116 to edit 3D spatial information or may utilize 3D spatial information for drawing and modeling.

The facility information acquisition unit 123 may obtain and process facility information corresponding to the road facility from the outside based on 3D location data belonging to the 3D spatial information and 2D location data included in the image data.

Specifically, the facility information acquisition unit 123 obtains a time-series aerial orthogonal image including facility information related to road facilities from the outside, extracts geometric line form information from the aerial orthogonal image, and Based on the information, it is possible to generate basic road information processed in the form of a document or data sheet in a time series of facility structure information and accessories. The road linear information may be information related to, for example, a road pavement as facility structure information, a ground condition, and a median divider as an accessory, a sign, and a trench-shaped side exit located on the side of the road.

The facility information acquisition unit 123 receives two-dimensional drawing data and 3D models accumulated in the form of images in relation to road facilities from outside, and digitalizes drawing data and 3D models through image-based object recognition and 3D model recognition. Drawing information can be created. The 3D model may be a rendered image and a schematic quantity related to the components of the road facility.

The facility information acquisition unit 123 acquires the current status/maintenance documents and data sheets accumulated in the form of images related to road facilities, and digitizes the document and the data sheet by image-based text recognition to obtain road status/maintenance information. Can be generated. The current status/maintenance document and data sheet may include the details of maintenance of road facilities, road registers, and statistical information using road status records.

Basic road information, drawing information, and road status/maintenance information are stored in the database unit 116 in association with road facilities, and the database unit 116 may transmit the above-described information to the requesting terminal according to an external request.

Meanwhile, the manipulation/display unit 118 may edit predetermined data according to a user request while providing image data for spatial information associated with object information to a user in the form of a screen. As shown in FIGS. 15 and 17, when the user designates at least a part of a specific road facility or maintenance part among the images displayed on the screen, the operation/display unit 118 provides 3D position data of a specific road facility and maintenance part. And control to display at least the name of the road facility among object information, or to edit at least one of 3D location data and object information by a user.

The image data expressed on the manipulation/display unit 118 is displayed on the screen as an RGB image or black and white images including 3D spatial information in an image pixel, and allows a user or a computing device to recognize an RGB image or a black and white image. .

In addition, when the user selects a part of the image as a point of interest, as shown in the right and lower drawings of FIG. 15, the area of interest screen generation unit 120 displays the area of interest in an enlarged screen format. The operation/display unit 118 can be controlled so as to be performed.

Meanwhile, the updater 122 acquires external data including object information including a plurality of attribute data, and extracts object information including at least metadata related to location data and detailed information of road facilities from the external data. , When at least one of a plurality of attribute data of object information of the spatial information image data recorded in the same location is different from the object information of external data, the spatial information image data may be updated with object information of external data. . The updated object information is stored in the database unit 116 in association with image data for spatial information.

The external data is a digital map or a road precision map, which is obtained from an external device other than the system 100 from the same location as the image data for spatial information of the system 100.

For example, as metadata belonging to object information linked to spatial information, the speed limit of a road sign is stored as 30km, but roads in the same location as metadata of external data such as digital maps or road precision maps. When external data in which the speed limit of the sign is recorded as 40 km is obtained, the update unit 122 changes object information of the image data for spatial information to 40 km.

The distortion removal unit 110, image data processing unit 112, object detection and recognition unit 114, database unit 116, update unit 122, and facility information acquisition unit 123 described with reference to FIG. It can be made in the high-load processing module (H), for example, the server side that needs processing. The operation/display unit 118 and the region of interest screen generation unit 120 may be implemented in a low-cost processing module L, for example, in the cloud or on the client side, which has a lower data processing burden than the above-described member. In the present embodiment, modules according to the data processing burden have been distinguished and described, but in other embodiments, all of the functions of the above-described members may be implemented in one integrated module.

Hereinafter, a method of implementing a road facility management solution based on a 3D-VR multi-sensor system according to an embodiment of the present invention will be described with reference to FIGS. 1 to 17.

First, the image sensor 102, the 3D survey sensor 104, and the navigation sensor 106 acquire each observation data and time information at the time of acquisition, and the data fusion unit 108 is used for each sensor 102 to 106 The internal geometry calculated based on the geometric parameter defined every time is obtained (S505).

The sensor data collection unit including the image sensor 102, the 3D survey sensor 104, and the navigation sensor 106 may additionally acquire unique data for each sensor 102 to 106. The internal geometry is described in detail in the description of the system 100 and detailed description thereof will be omitted.

Next, the data fusion unit 108 determines the observed data, the internal geometry, and the geometric relationship between the respective sensors 102 to 106 according to the position and posture of the platform 10 on which at least one of the sensors 102 to 106 is mounted. As shown in FIG. 6, data fusion for the geometric model is performed on the basis of the geometry information composed of the external geometry to be defined (S510). 6 is a diagram illustrating the concept of a geometric model for generating geometric structure information.

The data fusion unit 108 generates point cloud data related information related to the 3D survey sensor 104 based on observation data, geometric structure information, and time information, and is generated from the data of the image sensor 102 to generate geocoding image information. Image data including a may be generated. The point cloud data-related information and the geocoding image information may include the above-described data, and a detailed description thereof will be omitted.

Subsequently, the distortion removing unit 110 determines whether the image sensor is a wide area image sensor (S515). Image data from a wide area image sensor taken at each measurement point (circular point) as in FIG. 7(a) is generated as a distorted image as in FIG. 7(b). 7 is a diagram illustrating a distorted panoramic image as image data captured from a wide area image sensor.

The distortion removal unit 110 additionally depends on at least one of a type of a wide area image sensor, a good degree of distortion removal, a user's selection, and other options. An index-based distortion removal unit 110 and a geometric model-based distortion removal unit 110 may be selected.

Subsequently, when image data is generated from the wide area image sensor, the distortion removal unit 110 removes distortion of the image data (S415). The distortion removal process proceeds to the index-based distortion removal unit 110 or the geometric model-based distortion removal unit 110 selected according to at least one of a type of a wide area image sensor, a good degree of distortion removal, and a user's selection. Hereinafter, a distortion removal process will be described in detail by distinguishing between the index-based distortion removal unit 110 and the geometric model-based distortion removal unit 110 with reference to FIGS. 8 to 13.

8 is a flowchart illustrating a process of removing image distortion by an index-based distortion removing unit. 9 is a flowchart illustrating a process of removing distortion of an image by a geometric model-based distortion removing unit. 10 and 11 are diagrams schematically illustrating a process of removing image distortion by an index-based distortion removing unit. 12 is a diagram schematically illustrating a process of removing image distortion in a geometric model-based distortion removing unit. 13 is a diagram illustrating image data from which distortion is removed by a distortion removal unit.

First, the image distortion removal process in the index-based distortion removal unit 110 will be described with reference to FIGS. 8, 10 and 11, and image data for fisheye lenses distorted by a wide-area image sensor equipped with a fisheye lens is input. In the case of (S805), the pixel index assigning unit 128 assigns an index to each pixel to define a correspondence relationship between the distorted image data from the image data and the reference plane image data, as shown in FIG. 10 (S810). .

Next, the section decomposing unit 130 decomposes the distorted image data to which the index is assigned into a preset section (S815).

Subsequently, the planar image generator 132 refers to the reference planar image data, and generates planar image data by removing distortion of the distorted image data based on the distortion correction model of the decomposed section (S820).

The distortion correction model may be modeling based on an equation or index of the decomposed section.

If omnidirectional image data is input to the index-based distortion removal unit 110, the pixel indexing unit 128 and the section decomposing unit 130 process the image data, and the processing proceeds to the second step of FIG. 11.

Next, the section re-decomposition unit 134 re-decomposes the omnidirectional image decomposed by the section decomposition unit 130 into a preset section, as in the third step of FIG. 11. Subsequently, the planar image generation unit 132 removes the distortion based on a distortion correction model, such as an equation or an index, of the section re-decomposed by the section re-decomposition unit 134, and generates the planar image data as shown in the right figure of FIG. Generate.

On the other hand, the image distortion removal process in the geometric model-based distortion removal unit 110 will be described with reference to FIGS. 9 and 12, when the distorted omnidirectional image data is input by a wide area image sensor that generates an omnidirectional image ( S905), the 3D shape restoration unit 136 restores a virtual 3D shape from the distortion image data based on the camera geometry based on the distortion image data from the image data (S910).

Next, the planar image generation unit 138 defines a virtual camera geometry and generates planar image data by projecting it onto a virtual plane camera through the geometry (S915).

Referring back to FIG. 5, an image from which distortion is removed or an image without distortion (in the case of N in S515) is input to the image data processing unit 112, and the image data processing unit 112 includes pixels of the image data and Generated from point cloud data-related information (left drawing of Fig. 14) including at least three-dimensional position data by referring to indexes, geometry information, and time information given to define a correspondence relationship between three-dimensional position data of point cloud data-related information As shown in FIG. 14, the 3D spatial information 20 is registered in pixel units of the image data to generate image data for spatial information (S525).

The index generation will be described in detail, and detailed descriptions will be omitted.

A model for imparting 3D spatial information to pixels of image data may be defined as in Equation 1 described above, and some equations and variables may be modified or omitted during calculation.

When the image data is displayed in the form of an RGB image or a black and white image, the image data processing unit 112 provides each pixel with 3D spatial information or time, index, and navigation data to each pixel. Each time, geocoding image information, 3D spatial information, temporal information, indexes, and navigation data are all linked and stored in the database unit 116.

Next, the object detection and recognition unit 114 detects and recognizes object information related to the road facility and the maintenance part of the facility based on the image data for spatial information through machine learning (S530). Road facility and object information is generated by the facility recognition unit 140 and the repair part recognition unit 142 as shown in FIGS. 15 and 16. A plurality of attribute data and machine learning processes constituting object information are listed above, and detailed descriptions thereof will be omitted.

15 is a diagram illustrating a screen in which a part of the location data and object information of the road facility is output to the manipulation/display unit when a user designates a specific road facility in the image provided to the manipulation/display unit, and FIG. 16 is an object detection A diagram illustrating that the recognition unit detects and recognizes a maintenance part of a road facility based on image data for spatial information by machine learning, reads it as object information, and stores 3D maintenance section position data in the database unit.

The object detection and recognition unit 114 may perform preprocessing to improve the accuracy of object detection and recognition in image data for spatial information before machine learning, and a preprocessing technique will be omitted in detail.

Subsequently, the database unit 116 stores 3D map data including at least maintenance section location data based on image data for spatial information to which object information of road facilities and maintenance parts is added, as shown in the left drawing of FIG. 16. (S535).

At the same time, the database unit 116 stores in association with image data for spatial information, road facilities, and object information of a maintenance part.

Next, the manipulation/display unit 118 provides image data for spatial information associated with object information to the user in the form of a screen, and the region of interest screen generation unit 120 is used by the user as shown in the right and lower drawings of FIG. When a part of the image is selected as the region of interest, the manipulation/display unit 118 is controlled to display the region of interest in the form of an enlarged screen (S540).

As shown in FIGS. 15 and 17, when the user designates at least a part of a specific road facility or maintenance part among the images displayed on the screen, the operation/display unit 118 provides 3D position data of a specific road facility and maintenance part. And at least the name of the road facility among the object information.

According to the present embodiment, in order to easily and accurately check the status and maintenance of road facilities, a two-dimensional survey sensor 104 such as a laser sensor and a navigation sensor 106 are combined with the image sensor 102 as a center. An image pixel-based service can be provided by registering accurate 3D spatial information in each pixel of image data.

In addition, by defining the geometry of the system 100 around the image sensor 102, accuracy is improved and processing time is shortened. In addition, image-based machine learning can be applied to the system, reducing drawing and modeling work time and improving accuracy.

In addition, when the viewing angle between the 3D survey sensor 104 such as Lidar and the image sensor 102 does not match, the data of the navigation sensor 106 and the time information according to the observation of each sensor are used to display the image pixels. 3D spatial information can be given.

In addition, it is possible to shorten data processing time and reduce data capacity by providing an index to each sensor 102 to 106 and data and linking with each other.

Hereinafter, a process of updating object information of image data for spatial information in a system implementing a road facility management solution based on a 3D-VR multi-sensor system according to an embodiment of the present invention will be described with reference to FIGS. 1 and 18 I will do it.

For example, in this situation, the speed limit of a road sign is stored as 30 km as metadata belonging to object information linked to image data for spatial information, but it is stored in the same location as metadata of external data such as digital maps or road precision maps. External data in which the speed limit of a road sign that is present is recorded as 40 km may be acquired.

First, after the update unit 122 generates image data for spatial information to which object information is added as in step S530 or S535. External data including object information including a plurality of attribute data is acquired (S1805).

Next, the updater 122 extracts object information including at least metadata related to location data and detailed information of road facilities from external data (S1810).

Subsequently, when at least one of the plurality of attribute data of the object information of the spatial information image data recorded at the same location is different from the object information of the external data (S1815), the image data for spatial information is It is updated with object information of external data (S1820). The updated object information is stored in the database unit 116 in association with image data for spatial information.

In contrast, when the image data for spatial information and external data are the same, the update unit 122 maintains object information of the image data for spatial information (S1825).

Components constituting the system 100 shown in FIGS. 1, 3 and 4 or steps according to the embodiments shown in FIGS. 5, 8, 9, and 18 are in the form of a program for realizing the function Can be recorded on a computer-readable recording medium. Here, the computer-readable recording medium refers to a recording medium that can be read by a computer by storing information such as data or programs by electrical, magnetic, optical, mechanical, or chemical action. Examples of such recording media that can be separated from a computer include portable storage, flexible disks, magneto-optical disks, CD-ROMs, CD-R/Ws, DVDs, DATs, and memory cards. In addition, as recording media fixed to mobile devices and computers, there are solid state disks (SSDs), hard disks, and ROMs.

Further, in the above, even if all the constituent elements constituting the embodiments of the present invention have been described as being combined into one operation, the present invention is not necessarily limited to these embodiments. That is, within the scope of the object of the present invention, all of the constituent elements may be selectively combined and operated in one or more. In addition, although all of the components may be each implemented as one independent hardware, some or all of the components are selectively combined to provide a program module that performs some or all functions combined in one or a plurality of hardware. It may be implemented as a computer program having.

Although the present invention has been described in detail through exemplary embodiments above, a person of ordinary skill in the art to which the present invention pertains will realize that various modifications can be made to the above-described embodiments without departing from the scope of the present invention. I will understand. Therefore, the scope of the present invention is limited to the described embodiments and should not be determined, and should be determined by all changes or modifications derived from the claims and the concept of equality as well as the claims to be described later.

Claims

A sensor data collection unit for collecting observation data and time information on road facilities from an image sensor, a navigation sensor, and a 3D survey sensor that acquires 3D geographic data;

Geometry based on geometry information consisting of internal geometry calculated based on geometric parameters defined for each sensor and external geometry defining a geometric relationship between each sensor according to the position and attitude of the platform on which at least one of the sensors is mounted A data fusion unit that performs data fusion on the model;

An index assigned to define a correspondence relationship between pixels of image data from observation data of the image sensor and three-dimensional position data of point cloud data related information from observation data of the 3D survey sensor, the geometry information, and the time information Referring to, an image data processing unit for generating image data for spatial information by registering 3D spatial information generated from point cloud data related information including at least the 3D position data in pixel units of the image data;

An object detection and recognition unit detecting and recognizing object information related to the road facility based on the spatial information image data; And

A system for implementing a road facility management solution based on a 3D-VR multi-sensor system including a database unit that stores the spatial information image data and the object information in association with each other.
The method of claim 1,

The point group data-related information includes 3D location data, point group color information, point group class information whose type is estimated from the object extracted from the 3D location data, and laser intensity when the 3D survey sensor is a laser sensor. A system that implements a road facility management solution including at least one of information.
The method of claim 1,

The object detection and recognition unit detects and recognizes a maintenance part of the road facility along with the road facility based on the image data for spatial information through machine learning,

The object information is a system for implementing a road facility management solution including attribute data including metadata, shape, shape, and texture related to the type, name, and detailed information of the object in relation to the road facility and the maintenance part.
The method of claim 3,

The database unit implements a road facility management solution for storing 3D map data including at least repair section location data based on the spatial information image data linked with the object information.
The method of claim 1,

Further comprising a manipulation/display unit capable of editing predetermined data while providing the spatial information image data associated with the object information to a user in a screen form,

The operation/display unit controls to display at least the name of the road facility among 3D location data of the specific road facility and the object information when the user designates at least a part of a specific road facility among the images displayed on the screen. Or, controlling the user to edit at least one of the 3D location data and the object information,

When the user selects a part of the image as a region of interest, a system for implementing a road facility management solution further comprising a region of interest screen generator for controlling the manipulation/display unit to display the region of interest in an enlarged screen form .
The method of claim 5,

The image data processing unit, the object detection and recognition unit, and the database unit are implemented in a high-load processing module, and the display/manipulation unit is implemented in a low-load processing module having a lower computational processing capability than the high-load processing module. The system that implements the solution.
The method of claim 1,

Obtaining external data including object information having a plurality of attribute data, extracting object information including at least metadata related to location data and detailed information of the road facility from the external data, and recorded at the same location When at least one of the plurality of attribute data of object information of the spatial information image data is different from the object information of the external data, further comprising an update unit for updating the spatial information image data to the object information of the external data A system that implements a road facility management solution.
The method of claim 1,

When the image sensor is composed of a wide area image sensor having a wider field of view than a flat image sensor generating a flat image, image data from observation data of the wide area image sensor is generated as a panoramic image, and distortion of the panoramic image is reduced. Further comprising a distortion removing unit to remove,

The distortion removal unit includes an index-based distortion removal unit that performs index-based image distortion removal when the image data is input, and a geometric model-based distortion removal unit that performs image distortion removal based on a geometric model when the image data is input. ,

The distortion removal unit is a road for performing image distortion removal by one of the index-based distortion removal unit and the geometric model-based distortion removal unit according to at least one of a type of the wide area image sensor, a good level of distortion removal, and a user's selection A system that implements a facility management solution.
The method of claim 8,

The index-based distortion removing unit,

A pixel index giving unit for assigning an index to each pixel to define a correspondence relationship between the distorted image data from the image data and the reference plane image data;

A section decomposition unit that decomposes the distorted image data to which the index is assigned into a preset section,

A system for implementing a road facility management solution including a planar image generator configured to generate planar image data by removing distortion of the distorted image data based on a distortion correction model of a decomposed section with reference to the reference plane image data.
The method of claim 8,

The geometric model-based distortion removing unit 3D shape restoration unit for restoring a virtual 3D shape from the camera geometry-based distortion image data based on the distortion image data from the image data,

A system for implementing a road facility management solution including a planar image generator for generating planar image data by defining a virtual camera geometry and projecting it onto a virtual plane camera through the geometry.
The method of claim 1,

Further comprising a facility information acquisition unit for obtaining and processing facility information corresponding to the road facility from an outside based on the 3D location data belonging to the 3D spatial information and 2D location data included in the image data,

The facility information acquisition unit obtains a time series aerial orthogonal image including the facility information related to the road facility from the outside, extracts geometric line form information from the aerial orthogonal image, and uses the road linear information Based on the facility structure information and accessories in time series, it generates basic road information processed in the form of documents or data sheets,

The facility information acquisition unit receives two-dimensional drawing data and 3D model accumulated in image form in relation to the road facility from outside, and digitizes the drawing data and the 3D model by image-based object recognition and 3D model recognition. Create drawing information,

The facility information acquisition unit obtains the current status/maintenance document and data sheet accumulated in the form of an image in relation to the road facility, and generates road status/maintenance information by digitizing the document and the data sheet by image-based text recognition. ,

A system for implementing a road facility management solution that stores the basic road information, the drawing information, and the road status/maintenance information in the database unit in association with the road facility.
Collecting observation data and time information for road facilities, respectively, from an image sensor, a navigation sensor, and a 3D survey sensor that acquires 3D geographic data;

Geometry based on geometry information consisting of internal geometry calculated based on geometric parameters defined for each sensor and external geometry defining a geometric relationship between each sensor according to the position and attitude of the platform on which at least one of the sensors is mounted Performing data fusion for the model;

An index assigned to define a correspondence relationship between pixels of image data from observation data of the image sensor and three-dimensional position data of point cloud data related information from observation data of the 3D survey sensor, the geometry information, and the time information Generating image data for spatial information by registering 3D spatial information generated from point cloud data-related information including at least the 3D position data in pixel units of the image data;

Detecting and recognizing object information related to the road facility based on the spatial information image data; And

A method of implementing a road facility management solution based on a 3D-VR multi-sensor system comprising the step of linking and storing the spatial information image data and the object information.
The method of claim 12,

The point group data-related information includes 3D location data, point group color information, point group class information whose type is estimated from the object extracted from the 3D location data, and laser intensity when the 3D survey sensor is a laser sensor. A method of implementing a road facility management solution including at least one of information.
The method of claim 12,

The detecting and recognizing the object information includes detecting and recognizing a maintenance part of the road facility along with the road facility based on the image data for spatial information through machine learning,

The object information is a method for implementing a road facility management solution including attribute data including metadata related to the type, name, detailed information, shape, shape, and texture of the object in relation to the road facility and the maintenance part.
The method of claim 14,

In the linking and storing of the spatial information image data and the object information, the road facility management for storing 3D map data including at least repair section location data based on the spatial information image data linked with the object information How to implement the solution.
The method of claim 12,

After the step of linking and storing the spatial information image data and the object information,

Providing the image data for spatial information associated with the object information to a user in the form of a screen or editing predetermined data; And

When the user selects a part of the image as the region of interest, controlling to display the region of interest in an enlarged screen form,

When the user designates at least a part of a specific road facility among the images displayed on the screen, at least displays the name of the road facility among 3D location data of the specific road facility and the object information, or the 3D location data And a method for implementing a road facility management solution that controls a user to edit at least one of the object information.
The method of claim 12,

After the step of linking and storing the spatial information image data and the object information,

Obtaining external data including object information having a plurality of attribute data;

Extracting object information including at least metadata related to location data and detailed information of the road facility from the external data;

When at least one of a plurality of attribute data of object information of the spatial information image data recorded at the same location is different from object information of the external data, the spatial information image data is converted to the object information of the external data. A method of implementing a road facility management solution further comprising the step of updating.