KR20230072002A - System for updating spatial information using device - Google Patents

Abstract

A spatial information map updating system using a device is disclosed.
The spatial information map updating system may include a reference image database unit for storing reference image data in association with reference geometric information including first external geometric data; a search selection unit for selecting the image data having the highest degree of matching with the reference image data by referring to the 2-dimensional location data and the 3-dimensional location data of the image data of the device; a matching unit matching the reference image data and selected image data; a geometric conversion unit for geometrically transforming device geometric information of the selected image data and correcting the 3D device geometric information; an object information extractor extracting a specific type of object from 3D image information generated based on the image data of the device; a spatial information generating unit generating new spatial information including object information in the 3D image information based on the same object of the reference image data as the extracted object; and a spatial information updating unit determining whether the new spatial information is different from the existing spatial information in the object information of the same location, and updating the existing spatial information with the new spatial information if there is different object information. .

Description

Spatial information map update system using device {SYSTEM FOR UPDATING SPATIAL INFORMATION USING DEVICE}

The present disclosure relates to a spatial information map updating system using a device, and more particularly, to a spatial information map updating system capable of updating 3D spatial information substantially in real time using a device capable of obtaining an image.

In the field of spatial information, updating spatial information occupies a large portion of spatial map maintenance. In particular, the road operation and management business is a continuous work that requires occasional or annual maintenance and takes a long time. Existing maintenance methods require a lot of time and money as a lot of manpower directly visits the site to understand the current status and decides whether or not renewal is necessary.

Accordingly, by using smart devices equipped with an operation computing system along with video camera sensors, positioning sensors, geomagnetic sensors, gyro sensors, geomagnetic sensors, and communication modules such as smart phones, HMD (Head Mount Display), and smart glasses, A technique for generating or updating spatial information by obtaining local data has been attempted.

However, there is a difficulty in connecting information obtained from image data of a general single-camera-based smart device to a service based on 3D spatial information. This is due to problems with data capacity and computational complexity in utilizing 3D spatial information, and methods for overcoming these problems are continuously being proposed.

A technical problem of the present disclosure relates to a spatial information map updating system using a device, and more specifically, to provide a spatial information map updating system capable of updating 3D spatial information in real time using a device capable of acquiring images. It has a purpose.

The technical problem of the present disclosure is, in detail, spatial information map renewal that can construct 2D static data into 3D dynamic data, rapidly acquires data in a wide area, and dramatically reduces work time compared to data collection by manpower. Its purpose is to provide a system.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

According to one aspect of the present disclosure, a system for updating a spatial information map using a device includes reference image data stored in association with reference geometry information including first external geometry data having 3D position data and first posture data. an image database unit; Device geometry information including second external geometry data having at least two-dimensional positional data in relation to image data obtained from a device is acquired, and the reference image data and the best-in-class positional data are obtained by referring to the two-dimensional positional data and the three-dimensional positional data. a search selection unit that selects the image data having matching degree; a matching unit matching the reference image data and the selected image data based on feature geometries of a predetermined shape common between the reference image data and the selected image data; In the common feature geometries, a geometric conversion unit that performs geometric conversion of device geometry information of the selected image data with respect to the reference image data to include the 3D position data, and corrects the 3D device geometry information; a 3D image information generating unit generating 3D image information for image data of the device based on the 3D device geometry information and matching information on the feature geometry; an object information extractor extracting a specific type of object from the 3D image information; A space for generating new spatial information including object information related to the object stored in the 3D device geometry information and the reference image data in the 3D image information based on the object of the reference image data that is the same as the extracted object. information generating unit; and when the new spatial information is stored in the spatial information database unit and existing spatial information already stored in the spatial information database unit exists, the new spatial information is different from the existing spatial information in object information at the same location. and a spatial information updating unit that determines whether or not the object information is different as a result of the determination and updates the existing spatial information with the new spatial information.

According to another embodiment of the present disclosure, the first posture data may define a viewing direction set to generate a reference image.

According to another embodiment of the present disclosure, the second external geometric data further includes second attitude data defining a viewing direction set to generate the image,

The search selection unit searches for a candidate group of the image data that can be compared with the reference image data based on the 3D location data and the 2D location data, and selects the standard among the candidate groups with reference to the first and second posture data. The image data having the highest degree of matching with the image data may be selected.

According to another embodiment of the present disclosure, the reference geometric information further includes first internal geometric data based on a geometric parameter defined in a sensor set to generate the reference image, and the device geometric information is used to generate the image. Further including second internal geometric data based on a geometric parameter defined in the set image sensor, selection of the image data having the highest degree of matching in the search selection unit and the 3D device geometry information in the geometry conversion unit Correction of may be referenced by further including the first and second internal geometric data.

According to another embodiment of the present disclosure, when the reference image originates from a laser scanning sensor, the reference image uses virtual image data obtained by converting point cloud data obtained from the laser scanning sensor, and The 3D location data may be defined as 3D location data of the virtual image data, and the first posture data may be defined as a viewing direction of the virtual image data.

According to another embodiment of the present disclosure, feature geometries of image data associated with the 3D device geometry information are randomly sampled, and a distance between the sampled feature geometries and the corresponding feature geometries of the reference image data is minimized. A matching optimum value estimator for estimating the feature geometry of the sampling image data to be the matching optimum value, and the 3D device geometry information geometrically transformed in relation to the feature geometry estimated with the matching optimum value as the final 3D device geometry information. The apparatus may further include a device geometric information estimator for estimating, and among information based on generating the 3D image information, the final 3D device geometric information may be used as the 3D device geometric information.

The features briefly summarized above with respect to the disclosure are merely exemplary aspects of the detailed description of the disclosure that follows, and do not limit the scope of the disclosure.

According to the present disclosure, it relates to a spatial information map updating system using a device, and more specifically, it is possible to provide a spatial information map updating system capable of updating 3D spatial information in real time using a device capable of acquiring images. there is.

Specifically, according to the present disclosure, it is possible to construct 2-dimensional static data into 3-dimensional dynamic data, and rapidly acquire data in a wide area to drastically reduce work time compared to data collection by manpower.

Effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below. will be.

1 is a configuration diagram of a spatial information map updating system using a device according to an embodiment of the present disclosure and a device communicating therewith.
2 is a flowchart illustrating a process of generating spatial information of a device using 3D image information.
3 is a diagram showing an example of 3D image information stored in a reference database unit.
4 is a form in which the reference image database unit stores virtual image data obtained by converting point cloud data obtained from a laser scanning sensor.
5 is a diagram illustrating a process of searching for a candidate group of video data that can be matched with reference video data.
6 is a diagram illustrating a process of selecting image data having the highest degree of matching with reference image data.
7 is a diagram illustrating a process of calculating 2D position data of image data and 3D position data of reference image data corresponding thereto.
8 is a diagram showing an example in which the location and viewing direction of image data of a device are corrected with final 3D device geometry information.
9 is a flowchart illustrating a process of updating spatial information using a spatial information map updating system according to another embodiment of the present disclosure.
10 is a diagram illustrating an example of generating new space information including final 3D device geometry information and object information in 3D image information.
11 is a diagram showing an example of a process of updating spatial information.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

In describing the embodiments of the present disclosure, if it is determined that a detailed description of a known configuration or function may obscure the gist of the present disclosure, a detailed description thereof will be omitted. And, in the drawings, parts irrelevant to the description of the present disclosure are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, when a component is said to be "connected", "coupled" or "connected" to another component, this is not only a direct connection relationship, but also an indirect connection relationship where another component exists in the middle. may also be included. In addition, when a component "includes" or "has" another component, this means that it may further include another component without excluding other components unless otherwise stated. .

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements unless otherwise specified. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment. can also be called

In the present disclosure, components that are distinguished from each other are intended to clearly explain each characteristic, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form a single hardware or software unit, or a single component may be distributed to form a plurality of hardware or software units. Accordingly, even such integrated or distributed embodiments are included in the scope of the present disclosure, even if not mentioned separately.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, an embodiment composed of a subset of components described in one embodiment is also included in the scope of the present disclosure. In addition, embodiments including other components in addition to the components described in various embodiments are also included in the scope of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings.

Hereinafter, with reference to FIG. 1, an image having 3D image information is obtained by performing a predetermined process by matching image data of 2D location information obtained from a device according to an embodiment of the present invention with 3D image data of a reference image database unit. A spatial information generating and updating system of a device for generating data will be described.

1 is a configuration diagram of a spatial information map updating system using a device according to an embodiment of the present disclosure and a device communicating therewith.

The system 100 receives information in which image data of spatial objects such as terrain and features captured by the device 102 are combined with two-dimensional location data and the direction of the field of view of the image sensor 104 estimated at the time of capturing. do.

First, the device 102 will be described. The device 102 may include an image sensor 104, a positioning sensor 106, an IMU sensor (Inertial Measurement Unit; 106), a central processing unit 110, and a communication unit 112. there is.

The device 102 is a device that captures a surrounding environment while moving through a person or a moving object, and generates image data associated with two-dimensional location data and transmits/receives it to the system 100 through a wired/wireless network. For example, the device 102 may be a smart phone, a head mount display (HMD), smart glasses, a vehicle black box that recognizes vehicle collisions and accidents, etc., but is not limited thereto as long as the device implements the above functions.

The image sensor 104 may include a plurality of sensors mounted on the device 102 to acquire image data and time information for an image at the time when the image data was acquired by capturing an image of a surrounding object, for example, a terrain or a feature, as an image. can The image sensor 104 may be a planar image sensor, a wide area image sensor having a wider field of view than the planar image sensor, or a surveying or non-surveying camera, but is not limited thereto. The wide-area image sensor generates a stereo image in which images having different viewing directions are combined, or a panoramic image in which images of cameras disposed within a predetermined viewing angle are combined. For example, a camera having a fisheye lens to generate a fisheye lens image Alternatively, it may be a multi-camera system that outputs omnidirectional images by using multiple cameras.

When the image data originates from a wide-area image sensor that generates a distorted image, the image data is retrieved by a search selection unit 116 to be described later. Planar image data generated by converting the distorted image into planar image data may be used, and a detailed conversion process will be described later.

The positioning sensor 106 detects navigation information such as the positioning coordinates of an object and the location of the device 102 to obtain a two-dimensional position including two-dimensional positioning data related to the coordinates of the object and positioning time data at the time the position data was acquired. It is a sensor that acquires data. The positioning sensor 106 may be configured with, for example, a position acquisition device that acquires a moving position of the platform through a global positioning system (GPS) in which the Global Positioning System (GNSS) is used.

When the positioning sensor 106 uses GNSS, the position information basically includes positioning data and positioning time data, as well as the status and number of navigation satellite information, altitude information of the receiver, speed, positioning accuracy, true north information, etc. You can have it as auxiliary data.

The IMU sensor 108 detects the posture and speed of the device 102, and from this, the central processing unit 110 estimates posture data such as the viewing direction and angle of view of the image sensor 104, such as the shooting direction and angle. can do. The viewing direction of the image sensor 104 may be direction information shown in FIG. 10 as an azimuth measured by the IMU sensor 108, and may be an azimuth expressed as north, south, east, west, or the like of a predetermined coordinate system, or an azimuth angle determined relative to a movement path. can The viewing angle of the image sensor 104 may be generated by measuring an inclination of the image sensor 104 with respect to the ground by an accelerometer, an angular accelerometer, a gravimeter, etc. constituting the IMU sensor 108 . The IMU sensor may include a posture acquisition device that acquires or calculates posture data of the device 102 and an image sensor through an inertial navigation system (INS).

The central processing unit 110 converts the two-dimensional position data and posture data of the image sensor 104 obtained from the positioning sensor 106 and the IMU sensor 108 to the image data acquired from the image sensor 104 (hereinafter referred to as 'second Based on the geometric parameters defined in the image sensor 104, the positioning sensor 106, and the IMU sensor 108 together with external geometric data (hereinafter referred to as 'second external geometric data') having 'position data') Device geometry information including internal geometry data (hereinafter referred to as 'second internal geometry data') may be generated. The central processing unit 110 may control transmission of the above-described data through the communication unit 112 or may receive specific data from the system 100 and perform necessary processing.

The second external geometry data is based on a parameter value that varies whenever image information of the image sensor 104 is acquired due to the position and posture of the moving device 102, that is, the position and posture of the image sensor 104. It is the error of the calculated observation data, and in this case, it can be calculated through a predetermined equation or modeling. In the case where the image data originates from a wide-area image sensor, the 2D location data may utilize the 2D location data of the planar image data, and the second attitude data may be defined as the viewing direction and viewing angle of the planar image data.

The second internal geometric data is a unique value of the sensors 104 to 108 and is an error in observation data of each sensor 104 to 108 due to a parameter maintained regardless of whether the device 102 moves. The geometric parameter of the image sensor 104 may include any one or more of a focal length, a principal point location, a lens distortion parameter, and a sensor format size. The geometric parameters of the positioning sensor 106 or IMU sensor 108 may include one or more of an axial scale and an axial offset.

The system 100 includes a reference image database unit 114, a search selection unit 116, a matching unit 118, a geometric conversion unit 120, an optimal matching value estimation unit 122, and a device geometric information estimation unit 124. ), a 3D image information generating unit 126, an object information extracting unit 128, a spatial information generating unit 130, a spatial information updating unit 132, and a spatial information database unit 134.

The reference image database unit 114 is linked with reference geometry information including 3D positional data of the reference image and first external geometry data having first attitude data defining the viewing direction and viewing angle set to generate the reference image. Stores the reference image data. The 3D location data may be 2D location data, that is, XYZ coordinates obtained by adding an altitude value Z from XY coordinates. The viewing direction and viewing angle are substantially the same as those described for the image sensor 104 . In addition, the reference geometry information may further include first internal geometry data based on a geometry parameter defined in a sensor set to generate a reference image.

The reference image is obtained from the outside, is an image that includes at least 3D location data, and is processed into reference image data that can be compared with image data obtained from the device 102 . The sensor initially used to generate the reference image may be at least one of an image acquisition device having higher positioning accuracy and image realization performance than the device 102 and a laser scanning sensor (eg, Lidar). The reference geometric information associated with the reference image includes, for each pixel coordinate, 3D location information, color information such as RGB data, first external geometric data, and first internal geometric data, and the reference image combined with the reference geometric information It may be stored as reference image data.

When the sensor initially used to generate the reference image is an image sensor, the first internal geometric data is substantially the same as the image sensor 104 described in the second internal geometric data. When the sensor used initially is a laser scanning sensor, the first internal geometric data may be, for example, at least one of an incident angle of each laser, a distance scale, a distance direction offset, and an axial direction offset.

When the reference image originates from a wide-area image sensor that generates a distorted image, the reference image uses planar image data generated by converting the distorted image into planar image data, and the 3D position data is the 3D location of the planar image data. Data is utilized, and the first attitude data may be defined as a viewing direction and a viewing angle of the planar image data. A detailed conversion process will be described later.

In the case where the reference image originates from the laser scanning sensor, the reference image may use virtual image data obtained by converting point cloud data obtained from the laser scanning sensor. The 3D location data is the 3D location of the virtual image data. data, and the first posture data may be defined as a viewing direction of the virtual image data. The conversion of the point cloud data may be performed by virtual camera modeling taking into account the reflection intensity and altitude value of the point cloud data. When an image sensor is additionally involved in generating the reference image, color data acquired by the image sensor when converting point cloud data may be included in the virtual image data.

In addition, the reference image database unit 114 connects and stores information related to a specific type of object, that is, object information, together with the reference image data. For example, the object may correspond to a building that needs to be displayed on a map, a traffic sign that needs to be recognized for autonomous driving, a traffic light, a lane, and the like. In this case, the object information may include the type of the object, the overall shape, a specific geometric shape such as a line, plane, circle, or sphere constituting at least a part of the object, color, and texture-related properties.

On the other hand, the search selection unit 116 acquires device geometry information in relation to the image data acquired from the device 102, and first and second external geometry data including 3D and 2D position data and attitude data, A candidate group of image data that can be matched with the reference image data is searched with reference to the first and second internal geometric data, and image data having the highest degree of matching with the reference image data is selected from among the candidate groups.

Here, the image data may be used as a descriptor having characteristics capable of representing a specific object as well as data in a matrix form having color information.

Specifically, the search selection unit 116 searches for a candidate group of image data that can be compared with the reference image data based on the 3D position data and the 2D position data, and selects a reference image from among the candidate group by referring to the first and second posture data. Image data having the highest degree of matching with data may be selected.

When the image data acquired from the device 102 originates from the wide-area image sensor, the search selection unit 116 removes the distortion of the wide-area image and converts the image data into planar image data as described above. can be performed prior to the selection process.

In the above embodiment, the search selection unit 116 analyzes the first and second external geometric data including posture data and the first and second internal geometric data to select a candidate group of image data and image data having the highest degree of matching. Although it is described as selecting , a process of selecting a candidate group and image data may be performed based on values other than the altitude value of the 3D location data together with the 2D location data.

The matching unit 118 matches the reference image data and the selected image data based on feature geometries of a predetermined shape common between the reference image data and the selected image data, and through a geometric model such as a collinear conditional expression with reference to the feature geometry. A plurality of selected image data may be combined.

A feature geometry of a predetermined shape may be extracted based on a specific geometric shape of an object expressed in the selected image data and the reference image data estimated to have the same location. The object may be randomly selected as an object from which feature geometry such as a point, line, plane, column, etc. can be easily extracted from image data, such as a road sign, a uniquely shaped object, or a building boundary. This feature geometry adopts a predetermined shape that is highly likely to be estimated as the same object even if the attitude data between the selected image data and the reference image data are slightly different, and may be, for example, a lane marked on a road surface, a road sign, or a building outline.

In common feature geometries, the geometric transformation unit 120 geometrically transforms the geometric information of the device of the selected image data for the reference image data by geometric transformation modeling to include the 3D position data, and corrects the 3D device geometric information. do.

Geometric transformation modeling may use a collinear condition expression such as Equation 1 or substitute modeling for one of perspective transformation, affine transformation, and homography transformation that can reflect the geometry of image data in a matrix form.

[Equation 1]

Image coordinates are 2D positional data of image data related to the same feature geometry, and object coordinates are 3D positional data of reference image data for the same feature geometry. The camera position and posture are first external geometric data of the reference image data, and the principal point position, focal length, and lens distortion are first internal geometric data.

The matching optimal value estimator 122 randomly samples feature geometries of image data linked with 3D device geometry information, and the sampled image data for which the distance between the sampled feature geometries and the corresponding feature geometries of reference data is minimized. The feature geometry of can be estimated as the matching optimal value.

Specifically, a plurality of image data sharing feature geometries are sampled for each of the same estimated feature geometries, and at the same time, reference image data having feature geometries are interlocked and selected. Next, when the image data and reference image data sharing the same estimated feature geometry are arranged according to the 3D device geometry information, it is determined that the distance between the image data and the reference image data between the feature geometries of the same estimate is minimal. The feature geometries of the image data are estimated as optimal matching values.

The device geometric information estimator 124 may estimate the 3D device geometric information geometrically transformed in relation to the feature geometry estimated to be the optimal matched value as the final 3D device geometric information.

The 3D image information generator 126 generates 3D image information for the image data of the device 102 based on the final 3D device geometry information and matching information on feature geometry.

The 3D image information is the corrected second external image data including the image data corrected by geometric transformation and the 3D position data and corrected second posture data in connection with the image data, such as the reference image data of the reference image database unit 114. It may include geometry data, final 3D device geometry information including corrected second internal geometry data, color data, time data, and the like.

The object information extractor 128 may extract a specific type of object from 3D image information derived from image data.

Specifically, the object information extractor 128 may extract object information related to the attribute of the extracted object based on data included in the 3D image information.

The object information extractor 128 detects candidate group data related to the object identified from the image data included in the 3D image information, final 3D device geometry information, color data, time data, etc. using a machine learning technique, and detects the detected candidate group It is possible to recognize the information of an object by sequentially applying additional machine learning models to the data.

Machine learning is performed in two steps: object detection and object recognition. In the object object detection step, a candidate group is detected within the data through machine learning based on properties such as vectors such as shape, color, and texture of the object object. Then, additional machine learning models are sequentially applied to the detected candidate group to recognize and classify the information represented by the target object.

The spatial information generating unit 130 includes, based on the object of the reference image data identical to the extracted object, object information related to the object stored in the final 3D device geometry information and the reference image data in the 3D image information, a predetermined coordinate system, For example, new spatial information according to the absolute coordinate system may be generated, transmitted to and stored in the spatial information database unit 134 . Spatial information is spatial data described later, and detailed information will be described later.

The spatial information database unit 134 structures and stores observation data pre-obtained from heterogeneous sensors and pre-generated data for drawing as spatial data based on the observation data according to a predetermined coordinate system, for example, an absolute coordinate system. Spatial data is data generated from corresponding coordinates in a predetermined coordinate system, and includes observation data, internal geometry, external geometry, unique data, time information, acquisition environment information, separation amount due to positional error of sensor data between heterogeneous sensors, accuracy, and precision. It may include related probability information, point cloud data related information, image data related information, object information, and map related information. In this embodiment, spatial data is described as including all of the above data or information, but is not limited thereto, and may be set to include only some of the above data/information according to design specifications. The spatial information database unit 134 may be constructed as 3D dynamic data (XYZ, Time) by time-sequentially storing the same 3D spatial data.

When existing spatial information already stored in the spatial information database 134 exists, the spatial information updating unit 132 determines whether the new spatial information is different from the existing spatial information in the object information of the same location; As a result of the determination, with respect to different object information, existing spatial information may be updated with new spatial information.

According to the present embodiment, a 2D image matching method can be applied to matching between image data having 2D location data of the device 102 and reference image data having 3D location data, rather than a matching method between different dimensions. Therefore, the matching process can be performed quickly with low complexity.

In addition, it is possible to generate 3D image information by geometrically transforming and correcting the location data of the image data with 3D location data of high precision and accuracy included in the reference image data. It can be usefully utilized in subsequent data processing.

In addition, in this embodiment, final 3D device geometry information is generated with feature geometries having a minimum mutual distance between feature geometries estimated through the matching optimal value estimation unit 122, but the second external geometry data Since is data that can achieve good matching between image data, if the condition that the distance between feature geometries is less than or equal to a threshold value is satisfied, the geometry conversion unit 120 may be omitted.

Hereinafter, with reference to FIGS. 2 to 8 , a process of generating spatial information of a device using 3D image information will be described. In this embodiment, the first and second posture data are used in the process of selecting the image data having the highest degree of matching with the reference image data, and the function of the geometric conversion unit 120 is performed. However, as explained above, they may be omitted.

2 is a flowchart illustrating a process of generating spatial information of a device using 3D image information. 3 is a diagram showing an example of 3D image information stored in a reference database unit.

First, the search selection unit 116 acquires device geometry information including second external geometry data and second internal geometry data having two-dimensional position data and second attitude data related to image data obtained from the device 102. Do (S105).

The second external geometry data is based on a parameter value that varies whenever image information of the image sensor 104 is acquired due to the position and posture of the moving device 102, that is, the position and posture of the image sensor 104. It is the error of the calculated observation data, and in this case, it can be calculated through a predetermined equation or modeling. The second internal geometric data is a unique value of the sensors 104 to 108 and is an error in observation data of each sensor 104 to 108 due to a parameter maintained regardless of whether the device 102 moves. Geometric parameters for each of the sensors 104 to 108 are omitted from the above description.

Next, the search selection unit 116 includes at least three-dimensional position data and an image among reference geometry information of reference image data including first external geometry data having 3-dimensional position data and first posture data and first internal geometry data. A candidate group of image data is searched based on the 2D location data of the data (S110).

Here, the image data may be used as a descriptor having characteristics capable of representing a specific object as well as data in a matrix form having color information.

The reference image data is stored in the reference image database unit 114, and as shown in FIG. 3, the reference image data stores the above-described data including 3D position data of pixels displayed in dark colors. The 3D location data may be 2D location data, that is, XYZ coordinates obtained by adding an altitude value Z from XY coordinates. The first external geometric data includes the above-described first attitude data together with the 3D position data, and the first internal geometric data is based on geometric parameters defined in a sensor set to generate a reference image.

The reference image is obtained from the outside, is an image that includes at least 3D location data, and is processed into reference image data that can be compared with image data obtained from the device 102 . The sensor initially used to generate the reference image may be at least one of an image acquisition device having higher positioning accuracy and image realization performance than the device 102 and a laser scanning sensor (eg, Lidar). The reference geometric information associated with the reference image includes, for each pixel coordinate, 3D location information, color information such as RGB data, first external geometric data, and first internal geometric data, and the reference image combined with the reference geometric information It may be stored as reference image data.

When the sensor initially used to generate the reference image is an image sensor, the first internal geometric data is substantially the same as the image sensor 104 described in the second internal geometric data. When the sensor used initially is a laser scanning sensor, the first internal geometric data may include, for example, any one or more of an incident angle of each laser, a distance scale, a distance direction offset, and an axial direction offset.

When the reference image originates from a wide-area image sensor that generates a distorted image, the reference image uses planar image data generated by converting the distorted image into planar image data, and the 3D position data is the 3D location of the planar image data. Data is utilized, and the first attitude data may be defined as a viewing direction and a viewing angle of the planar image data.

In the case where the reference image originates from the laser scanning sensor, the reference image may use virtual image data obtained by converting point cloud data obtained from the laser scanning sensor. The 3D location data is the 3D location of the virtual image data. data, and the first posture data may be defined as a viewing direction of the virtual image data.

As shown in FIG. 4 , the conversion of point cloud data may be performed by virtual camera modeling taking into consideration reflection intensity, altitude value, etc. of point cloud data located below the drawing. 4 is a form in which the reference image database unit stores virtual image data obtained by converting point cloud data obtained from a laser scanning sensor. As a result, virtual image data may be generated as in the above example of FIG. 4 . When an image sensor is additionally involved in generating the reference image, color data acquired by the image sensor when converting point cloud data may be included in the virtual image data.

As described above, the search for image data candidates based on the 3D location data of the reference image data and the 2D location data of the image data performed by the search selection unit 116 may be performed through the same search process as shown in the steps below in FIG. 5 . there is. 5 is a diagram illustrating a process of searching for a candidate group of video data that can be matched with reference video data.

Next, the search selection unit 116 selects the image data having the highest degree of matching with the reference image data from among the candidate groups by referring to the 2D location data, the 3D location data, and the first and second posture data (S115).

Specifically, if the first and second attitude data of the first and second external geometry data are different according to the viewing direction and viewing angle of the image data having the same or very similar two-dimensional position data as the reference image data, due to the image geometry Even the same feature geometries are recognized as different geometries, making it difficult to match the reference image data and image data. Accordingly, by selecting only image data having similar first and second posture data within a predetermined range at the same location from the candidate group, accuracy and processing speed of the subsequent matching step may be improved. In this example, as shown in FIG. 5, a search based on direction information is performed, and as shown in FIG. 6, image data adjacent to the location and posture of the reference image data and a predetermined range is finally selected. 5 is a diagram illustrating a process of searching for a candidate group of image data that can be matched with reference image data, and FIG. 6 is a diagram illustrating a process of selecting image data having the highest degree of matching with reference image data.

Subsequently, the matching unit 118 matches the reference image data and the selected image data based on feature geometries of a predetermined shape common between the selected image data and the reference image data (S120).

The matching unit 118 may combine a plurality of selected image data through a geometric model such as a collinear conditional expression with reference to feature geometries.

A feature geometry of a predetermined shape may be extracted based on a specific geometric shape of an object expressed in the selected image data and the reference image data estimated to have the same location. The object may be randomly selected as an object from which feature geometry such as a point, line, plane, column, etc. can be easily extracted from image data, such as a road sign, a uniquely shaped object, or a building boundary. This feature geometry adopts a predetermined shape that is highly likely to be estimated as the same object even if the attitude data between the selected image data and the reference image data are slightly different, and may be, for example, a lane marked on a road surface, a road sign, or a building outline.

Subsequently, the geometric transformation unit 120 geometrically transforms the device geometry information of the selected image data for the reference image data to include 3D position data by geometric transformation modeling in common feature geometries, thereby performing 3D device geometry information. Corrected by (S125).

Geometric transformation modeling may use a collinear condition expression such as Equation 1 above, or alternative modeling may be used for one of perspective transformation, affine transformation, and homography transformation that can reflect the geometry of image data in a matrix form.

7 is a diagram illustrating a process of calculating 2D position data of image data and 3D position data of reference image data corresponding thereto. In the case of using the collinear condition expression, in the same feature geometry, the image data is geometrically transformed into the 3D coordinates of the reference image data and corrected. 8 is a diagram showing an example in which the location and viewing direction of image data of a device are corrected with final 3D device geometry information. The dotted line is the 2D location data of the image data acquired from the device 102, and the solid line is the device 102 as the 2D coordinates are moved to a more accurate location while being corrected with 3D location data by geometric transformation correction for the image data. It indicates that conversion is performed with higher precision than the precision of the positioning sensor 106 of .

Next, the matching optimal value estimator 122 randomly samples the feature geometries of the image data associated with the 3D device geometry information so that the distance between the sampled feature geometries and the corresponding feature geometries of the reference data is minimized. The feature geometry of the sampled image data is estimated as an optimal matching value (S130). A specific process of matching optimal values will be omitted in detail.

Subsequently, the device geometric information estimator 124 estimates the geometrically transformed 3D device geometric information in relation to the feature geometry estimated to be the optimal matched value as the final 3D device geometric information (S135).

Next, the 3D image information generating unit 126 generates 3D image information for the image data of the device 102 based on the final 3D device geometry information and matching information on feature geometry (S140).

The 3D image information is the corrected second external image data including the image data corrected by geometric transformation and the 3D position data and corrected second posture data in connection with the image data, such as the reference image data of the reference image database unit 114. It may include geometry data, final 3D device geometry information including corrected second internal geometry data, color data, time data, and the like.

According to this embodiment, image data of 2D location information acquired by the low-performance image sensor 104, positioning sensor 106, etc. of the device 102 is 3D image information generated from a platform composed of high-performance sensors. 3D image information of convenience and high precision can be generated from image data obtained with a low-cost device by performing geometric transformation by matching with .

Hereinafter, a method for updating spatial information using a spatial information map updating system according to another embodiment of the present disclosure will be described with reference to FIGS. 9 to 11 .

9 is a flowchart illustrating a process of updating spatial information using a spatial information map updating system according to another embodiment of the present disclosure. 10 is a diagram illustrating an example of generating new space information including final 3D device geometry information and object information in 3D image information. 11 is a diagram showing an example of a process of updating spatial information.

First, the object information extractor 128 extracts a specific type of object from 3D image information derived from image data (S205).

Specifically, the object information extractor 128 may extract object information related to the attribute of the extracted object based on data included in the 3D image information.

The object information extractor 128 detects candidate group data related to the object identified from the image data included in the 3D image information, final 3D device geometry information, color data, time data, etc. using a machine learning technique, and detects the detected candidate group It is possible to recognize the information of an object by sequentially applying additional machine learning models to the data.

Next, the spatial information generator 130 includes object information related to the object stored in the final 3D device geometry information and the reference image data in the 3D image information based on the object of the reference image data that is the same as the extracted object. New spatial information according to a predetermined coordinate system, for example, an absolute coordinate system, is generated (S210).

The spatial information generator 130 may transmit and store new spatial information in the spatial information database 134 . Spatial information is data generated from corresponding coordinates of a predetermined coordinate system, and includes observation data, internal geometry, external geometry, unique data, time information, acquisition environment information, separation amount due to positional error of sensor data between heterogeneous sensors, accuracy, and precision. It may include related probability information, point cloud data related information, image data related information, object information, and map related information. In this embodiment, spatial information is described as including all of the above-described data or information, but is not limited thereto, and may be set to include only some of the above-described data/information according to design specifications.

As shown in FIG. 10, new space information is created by including final 3D device geometry information and object information in 3D image information, and based on this, a map on the right side can be created through a drawing process.

Next, if there is existing spatial information already stored in the spatial information database unit 134, the spatial information updating unit 132 determines whether the new spatial information is different from the existing spatial information in the object information of the same location. It is determined (S215), and as a result of the determination, for the different object information, as shown in FIG. 11, the existing spatial information is updated with new spatial information (S225). As a result of the determination, when object information is the same, existing spatial information may be maintained (S220).

Exemplary methods of this disclosure are presented as a series of operations for clarity of explanation, but this is not intended to limit the order in which steps are performed, and each step may be performed concurrently or in a different order, if desired. In order to implement the method according to the present disclosure, other steps may be included in addition to the exemplified steps, other steps may be included except for some steps, or additional other steps may be included except for some steps.

Various embodiments of the present disclosure are intended to explain representative aspects of the present disclosure, rather than listing all possible combinations, and matters described in various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented by a processor (general processor), controller, microcontroller, microprocessor, or the like.

The scope of the present disclosure is software or machine-executable instructions (eg, operating systems, applications, firmware, programs, etc.) that cause operations according to methods of various embodiments to be executed on a device or computer, and such software or It includes a non-transitory computer-readable medium in which instructions and the like are stored and executable on a device or computer.

Claims (6)

In the spatial information map updating system using a device,
a reference image database unit that stores reference image data in association with reference geometry information including first external geometry data having 3D position data and first attitude data;
Device geometry information including second external geometry data having at least two-dimensional positional data in relation to image data obtained from a device is acquired, and the reference image data and the best-in-class positional data are obtained by referring to the two-dimensional positional data and the three-dimensional positional data. a search selection unit that selects the image data having matching degree;
a matching unit matching the reference image data and the selected image data based on feature geometries of a predetermined shape common between the reference image data and the selected image data;
In the common feature geometries, a geometric conversion unit that performs geometric conversion of device geometry information of the selected image data with respect to the reference image data to include the 3D position data, and corrects the 3D device geometry information;
a 3D image information generating unit generating 3D image information for image data of the device based on the 3D device geometry information and matching information on the feature geometry;
an object information extractor extracting a specific type of object from the 3D image information;
A space for generating new spatial information including object information related to the object stored in the 3D device geometry information and the reference image data in the 3D image information based on the object of the reference image data that is the same as the extracted object. information generating unit; and
When the new spatial information is stored in the spatial information database unit and existing spatial information already stored in the spatial information database unit exists, whether the new spatial information is different from the existing spatial information in the object information of the same location. A spatial information map updating system comprising: a spatial information updating unit that determines whether object information is different as a result of the determination and updates the existing spatial information with the new spatial information.
According to claim 1,
The spatial information map updating system of claim 1 , wherein the first attitude data defines a viewing direction set to generate a reference image.
According to claim 2,
The second external geometry data further includes second attitude data defining a viewing direction set to generate the image,
The search selection unit searches for a candidate group of the image data that can be compared with the reference image data based on the 3D location data and the 2D location data, and selects the standard among the candidate groups with reference to the first and second posture data. A spatial information map updating system that selects the image data having the highest degree of matching with image data.
According to claim 3,
The reference geometric information further includes first internal geometric data based on a geometric parameter defined in a sensor set to generate the reference image, and the device geometry information is based on a geometric parameter defined in an image sensor set to generate the image. Further including second internal geometry data, selection of the image data having the highest matching degree in the search selection unit and correction to the 3D device geometry information in the geometry conversion unit are performed by the first and second Spatial information map updating system, further including internal geometric data for reference.
According to claim 1,
When the reference image originates from a laser scanning sensor, the reference image uses virtual image data obtained by converting point cloud data acquired from the laser scanning sensor, and the 3D position data is the virtual image data. , and the first attitude data is defined as a viewing direction of the virtual image data.
According to claim 1,
The feature geometries of the image data associated with the 3D device geometry information are randomly sampled, and the feature geometries of the sampled image data that minimize the distance between the sampled feature geometries and the corresponding feature geometries of the reference image data are matched. a matching optimum value estimator for estimating an optimum value; and
Further comprising a device geometric information estimator for estimating 3-dimensional device geometric information geometrically transformed in relation to the feature geometry estimated as the matched optimum value as final 3-dimensional device geometric information,
Among the information based on the generation of the 3D image information, the 3D device geometry information uses the final 3D device geometry information.

Images