KR20220062709A - System for detecting disaster situation by clustering of spatial information based an image of a mobile device and method therefor - Google Patents

Abstract

Provided are a disaster situation recognition system and method by spatial information clustering based on a mobile device image. The system comprises: a reference image database unit for storing reference image data in association with reference geometry information including three-dimensional positional data and first posture data; a search and selection unit for obtaining device geometry information including at least two-dimensional positional data in relation to image data obtained from a mobile device and selecting the image data having the highest degree of matching with the reference image data by referring to the two-dimensional positional data and the three-dimensional positional data; a matching unit for matching the reference image data and the selected image data; a geometric conversion unit for geometrically transforming the device geometric information of the selected image data matched with the reference image data by geometric conversion modeling and correcting the device geometric information into three-dimensional device geometric information; a three-dimensional image information generation unit for generating three-dimensional image information of the image data of the mobile device; a disaster type extraction unit for determining whether a disaster has occurred in the surrounding environment from the three-dimensional image information and classifying the disaster type for each disaster area; and a disaster information registration unit for registering location information related to the disaster area and the disaster type in association with the three-dimensional image information. Accordingly, disaster information with reliability can be generated from three-dimensional image information with high precision generated based on the image data of the mobile device.

Description

System for detecting disaster situation by clustering of spatial information based an image of a mobile device and method therefor}

The present invention relates to a disaster situation recognition system and method, and more particularly, to a disaster situation recognition system and method by spatial information clustering based on a mobile device image.

Smart mobile devices equipped with computational computing systems as well as video camera sensors, positioning sensors, geomagnetic sensors, gyro sensors, geomagnetic sensors, and communication modules such as smartphones, HMD (Head Mount Display), and smart glasses are being used for various purposes.

For example, in the case of navigation, vehicle location information is extracted by fusion of existing road network information and GPS location data, and driving information is provided to the starting point and destination based on the vehicle's movement route using road network information and GPS location data. delivered to the user.

As such, in constructing spatial information, the positional accuracy obtained from the observation sensor acts as a very important factor. However, smart devices have limitations in that they have no choice but to use low-performance sensors due to problems such as size, power, and price, which causes many errors.

And recently, as smart city, autonomous driving, and digital twin have begun to be emphasized as important technologies in the industry, there are limitations as a form of service based on existing 2D positioning information.

Therefore, although the field of application of 3D image information is increasing, there is a limit to obtaining accurate dimensional positioning information with a low-cost positioning sensor.

Data such as conventional road networks, CAD drawings, and maps were available in the existing navigation, but in machine learning applications, accurate surrounding spatial information was not recognized and sufficient information was not provided in response.

In addition, since the accuracy and precision of conventional road networks and CAD drawings were generated at the metric level, there were limitations in providing services such as autonomous driving and AR/VR that require performance beyond that. In order to overcome this, there is a need to expand to three-dimensional spatial information rather than two-dimensional reference spatial information.

However, it is difficult to connect information obtained from image data of a general single camera-based smart device to a service based on 3D spatial information. To this end, various techniques including Simultaneous Location And Mapping (SLAM) have been developed, but these techniques require a high-performance computing system.

If the above-mentioned system is applied to a mobile device equipped with a low-performance positioning sensor to recognize or monitor various local situations, especially disaster situations, it will be very burdensome in terms of cost and resources, and the reliability of local disaster information will be lowered. it can only be

Accordingly, various methods for acquiring accurate positioning information and disaster information based on image data acquired from a low-performance mobile device are being attempted.

The technical problem to be achieved by the present invention is to match image data of two-dimensional position information acquired by a low-performance image sensor of a mobile device, a positioning sensor, etc. with three-dimensional image information generated by a platform composed of high-performance sensors, and perform geometric transformation, etc. For spatial information clustering based on mobile device images, in which disaster information of the surrounding environment based on 3D image information can be built with high reliability by performing correction through It is to provide a disaster situation awareness system and method by

Objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

According to one aspect of the present invention for achieving the above technical problem, a disaster situation recognition system by spatial information clustering based on a mobile device image is a first posture defining a viewing direction set to generate three-dimensional position data and a reference image A reference image database unit for storing reference image data in association with reference geometry information including first external geometric data having data, and second external geometric data having at least two-dimensional position data in relation to image data acquired from a mobile device Obtaining device geometry information including, and searching for a candidate group of the image data that can be matched with the reference image data with reference to the 2D position data and the 3D position data, and the best matching with the reference image data among the candidate group a search and selection unit for selecting the image data having a diagram; , in common feature geometries, the geometric information of the device of the selected image data matched with the reference image data includes the three-dimensional position data, by geometric transformation of the geometric information of the device by geometric transformation modeling A geometric transformation unit for correcting the device geometry information, and a 3D image information generating unit for generating 3D image information with respect to the image data of the mobile device based on the matching information for the 3D device geometry information and the feature geometry And, from the 3D image information based on the image data to the mobile device, a disaster type extractor for determining whether a disaster has occurred in the surrounding environment of the mobile device and classifying the disaster type for each disaster occurrence region, and the disaster and a disaster information registration unit for registering location information related to an occurrence area and the type of disaster in association with the 3D image information.

In another embodiment, the disaster type extractor may classify the disaster type while determining whether the disaster has occurred through machine learning on the 3D image information.

In another embodiment, the disaster information registration unit may process the 3D image information by performing clustering based on a disaster type of the disaster occurrence area and geographic proximity between the disaster occurrence areas.

In another embodiment, the second external geometric data further includes second posture data defining a viewing direction set to generate the image, and the search selection unit is based on the 3D position data and the 2D position data Thus, a candidate group of the image data that can be contrasted with the reference image data may be searched, and the image data having the highest degree of matching with the reference image data may be selected from the candidate group with reference to the first and second posture data.

In addition, the reference geometric information further includes first internal geometric data based on a geometric parameter defined in a sensor configured to generate the reference image, wherein the device geometric information is a geometric parameter defined in an image sensor configured to generate the image Further comprising second internal geometric data based on The second internal geometric data may be further included for reference.

In another embodiment, when the reference image originates from a laser scanning sensor, the reference image uses virtual image data obtained by converting point cloud data obtained from the laser scanning sensor, and the 3D position data may be defined as 3D position data of the virtual image data, and the first posture data may be defined as a viewing direction of the virtual image data.

In another embodiment, by randomly sampling feature geometries of image data to which the 3D device geometry information is linked, a sampling image in which an amount of mutual separation between the sampled feature geometries and the corresponding feature geometries of the reference image data is minimized A matching optimal value estimator for estimating the feature geometry of the data as a matching optimal value, and a device geometry for estimating the geometrically transformed three-dimensional device geometry information in relation to the feature geometry estimated as the matching optimal value as the final three-dimensional device geometry information The device may further include an information estimator, wherein the final 3D device geometry information may be used as the 3D device geometry information among information based on generation of the 3D image information.

According to an aspect of the present invention for achieving the above technical problem, a disaster situation recognition method by spatial information clustering based on a mobile device image is a second external geometry having at least two-dimensional location data related to image data obtained from a mobile device. Obtaining device geometry information including data, and the three-dimensional position data among reference geometry information including three-dimensional position data linked to the reference image data and first external geometry data defining a viewing direction set to generate the reference image and a search selection step of searching for a candidate group of the image data that can be matched with the reference image data by referring to the two-dimensional position data, and selecting the image data having the highest degree of matching with the reference image data from among the candidate group; A matching step of matching the reference image data and the selected image data based on a predetermined type of feature geometry common between the reference image data and the selected image data, and the device of the selected image data matched with the reference image data A geometric transformation step of geometrically transforming the geometric information of the device by geometric transformation modeling so that the geometric information includes the three-dimensional position data and correcting it into three-dimensional device geometric information, and the three-dimensional device geometric information and the feature geometry 3D image information generating step of generating 3D image information with respect to the image data of the mobile device based on the matching information for the mobile device, and from the 3D image information based on the image data to the mobile device, Disaster type extraction step of determining whether a disaster has occurred in the surrounding environment and classifying disaster types for each disaster occurrence area, and a disaster of registering location information related to the disaster area and the disaster type in connection with the 3D image information information registration step.

The details of other embodiments are included in the detailed description and drawings.

According to the present invention, image data of two-dimensional position information acquired by a low-performance image sensor or positioning sensor of a mobile device is matched with three-dimensional image information generated from a platform composed of high-performance sensors, and corrected through geometric transformation, etc. By performing , it is possible to generate reliable disaster information from high-precision 3D image information generated based on image data of a mobile device.

Based on the 3D image information based on the mobile device, it is possible to accurately identify the accurate positioning information and the disaster type of the disaster area, and by clustering the disaster area by type of disaster and visualizing it on a map, etc., it will contribute to systematically checking the disaster situation. can

In addition, the effect derived through the configuration that can be understood by those skilled in the art through the present specification is not excluded.

1 is a configuration diagram of a system for recognizing a disaster situation by spatial information clustering based on a mobile device image and a mobile device communicating therewith according to an embodiment of the present invention.
2 is a flowchart illustrating a disaster situation recognition method by spatial information clustering based on a mobile device image according to another embodiment of the present invention.
3 is a diagram illustrating an example of 3D image information stored in a reference database unit.
4 is a flowchart illustrating a process of converting image data acquired from a wide area image sensor into flat image data by removing distortion.
5 is a diagram schematically illustrating a process of image distortion removal and flat image data conversion.
6 is a diagram in which the reference image database unit stores virtual image data obtained by converting point cloud data obtained from a laser scanning sensor.
7 is a diagram illustrating a process of searching for a candidate group of image data that can be matched with reference image data.
8 is a diagram illustrating a process of selecting image data having the highest degree of matching with reference image data.
9 is a diagram illustrating a process of calculating two-dimensional position data of image data and three-dimensional position data of reference image data corresponding to the two-dimensional position data of image data.
10 is a diagram illustrating an example in which a position and a viewing direction of image data of a mobile device are corrected with final 3D device geometry information.
11 is a diagram illustrating a process of machine learning for classification by disaster type.
12 is a diagram illustrating an example of labeling a disaster type for each disaster location in a disaster occurrence area.
13 is a diagram illustrating an example of performing clustering for each disaster type in a disaster occurrence area.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. However, the present disclosure may be embodied in several different forms and is not limited to the embodiments described herein.

In describing an embodiment of the present disclosure, if it is determined that a detailed description of a well-known configuration or function may obscure the gist of the present disclosure, a detailed description thereof will be omitted. And, in the drawings, parts not related to the description of the present disclosure are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, when a component is "connected", "coupled" or "connected" to another component, it is not only a direct connection relationship, but also an indirect connection relationship in which another component exists in the middle. may also include. In addition, when a component is said to "include" or "have" another component, it means that another component may be further included without excluding other components unless otherwise stated. .

In the present disclosure, terms such as first, second, etc. are used only for the purpose of distinguishing one component from other components, and do not limit the order or importance between the components unless otherwise specified. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment is referred to as a first component in another embodiment. can also be called

In the present disclosure, the components that are distinguished from each other are for clearly explaining each characteristic, and the components do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form one hardware or software unit, or one component may be distributed to form a plurality of hardware or software units. Accordingly, even if not specifically mentioned, such integrated or distributed embodiments are also included in the scope of the present disclosure.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Accordingly, an embodiment composed of a subset of components described in an embodiment is also included in the scope of the present disclosure. In addition, embodiments including other components in addition to components described in various embodiments are also included in the scope of the present disclosure.

Also, in the present specification, the network may be a concept including both wired and wireless networks. In this case, the network may mean a communication network in which data exchange between the device and the system and devices can be performed, and is not limited to a specific network.

In addition, in the present specification, a device may be a mobile device such as a smart phone, a tablet PC, a wearable device, a laptop, and a head mounted display (HMD), as well as a fixed device such as a PC or a home appliance having a display function. Also, as an example, the device may be a computing device that can operate as a server, a vehicle, or an Internet of Things (IoT) device. That is, in the present specification, a device may refer to devices capable of performing the method according to the present invention, and is not limited to a specific type.

Hereinafter, with reference to FIG. 1, by performing a predetermined process by matching the image data of the two-dimensional position information acquired from the mobile device according to the embodiment of the present invention with the three-dimensional image data of the reference image database unit, having three-dimensional image information A disaster situation recognition system by spatial information clustering based on a mobile device image for generating image data and extracting or generating disaster information of the surrounding environment from 3D image information will be described.

1 is a configuration diagram of a system for recognizing a disaster situation by spatial information clustering based on a mobile device image and a mobile device communicating therewith according to an embodiment of the present invention.

The system 100 collects information in which image data of spatial objects such as terrain and features captured by the mobile device 102 are combined with two-dimensional position data and the viewing direction of the image sensor 104 estimated at the time of shooting. receive

First, the mobile device 102 is described. The mobile device 102 includes an image sensor 104 , a positioning sensor 106 , an IMU sensor (Inertial Measurement Unit) 106 , a central processing unit 110 , and a communication unit 112 . can do.

The mobile device 102 is a device that captures a surrounding environment while moving through a person, a moving object, etc., and is a device that generates image data associated with two-dimensional location data and transmits and receives image data to the system 100 through a wired/wireless network. For example, the mobile device 102 may be a smart phone, a Head Mount Display (HMD), smart glasses, a vehicle black box that recognizes vehicle collisions and accidents, etc., but is not limited thereto .

The image sensor 104 is mounted on the mobile device 102 and includes a plurality of sensors that acquire image data and time information for an image at the time the image data was acquired by photographing a surrounding object, for example, a terrain or a feature, as an image. can do. The image sensor 104 may be a flat image sensor, a wide-area image sensor having a wider field of view than a flat image sensor, or a surveying or non-surveying camera, but is not limited thereto. The wide-area image sensor generates a stereo image in which images having different viewing directions are combined, or a panoramic image in which an image of a camera disposed within a predetermined viewing angle is combined, for example, a camera having a fisheye lens to generate a fisheye lens image Or it may be a multi-camera system that outputs an omnidirectional image by a multi-camera using a plurality of cameras.

In the case where the image data originates from a wide-area image sensor that generates a distorted image, the image data is retrieved by the search selection unit 116, which will be described later. Planar image data generated by converting a distorted image into planar image data may be used, and the detailed conversion process will be described later.

The positioning sensor 106 detects navigation information such as the positioning coordinates of the object and the location of the mobile device 102, two-dimensional positioning data related to the object coordinates, and positioning time data at the time the position data was acquired. It is a sensor that acquires position data. The positioning sensor 106 may be constituted by, for example, a position acquiring device that acquires a moving position of the platform through a global positioning system (GPS) using the global global positioning system (GNSS).

When the positioning sensor 106 uses GNSS, the location information basically includes positioning data and positioning time data, as well as the status and number of navigation satellite information, altitude information of the receiver, speed, positioning precision, true north information, etc. You can have it as auxiliary data.

The IMU sensor 108 detects through the posture and speed of the mobile device 102, and from this, the image sensor 104 transmits posture data such as a viewing direction such as a photographing direction and an angle, and a viewing angle by the central processing unit 110. can be estimated The viewing direction of the image sensor 104 may be the direction information shown in FIG. 10 as an azimuth measured by the IMU sensor 108, etc. can The viewing angle of the image sensor 104 may be generated by measuring the inclination of an accelerometer, an angular accelerometer, a gravimeter, etc. constituting the IMU sensor 108 with respect to the ground of the image sensor 104 . The IMU sensor may be configured as an attitude acquisition device that acquires or calculates attitude data of the mobile device 102 and the image sensor through an inertial navigation system (INS).

The central processing unit 110 includes the image data obtained from the image sensor 104, two-dimensional position data and posture data (hereinafter, 'second Based on the geometric parameters defined in the image sensor 104 , the positioning sensor 106 , and the IMU sensor 108 together with external geometric data (hereinafter, referred to as 'second external geometric data') having 'posture data') Device geometry information including internal geometric data (hereinafter, referred to as 'second internal geometric data') may be generated. The central processing unit 110 may control to transmit the above-described data through the communication unit 112 , or may receive specific data from the system 100 and perform necessary processing.

The second external geometric data is based on a parameter value that is changed whenever image information of the image sensor 104 is acquired due to the position and posture of the mobile device 102 in motion, that is, the position and posture of the image sensor 104 . This is the error of the observed data calculated by doing this, and in this case, it may be calculated through a predetermined formula or modeling. When the image data originates from the wide-area image sensor, the 2D position data may use 2D position data of the planar image data, and the second posture data may be defined as a viewing direction and a viewing angle of the planar image data.

The second internal geometric data is an eigenvalue of the sensors 104 to 108 itself, and is an error of observation data for each sensor 104 to 108 due to a parameter maintained regardless of whether the mobile device 102 moves. The geometric parameter of the image sensor 104 may include any one or more of a focal length, a principal point position, a lens distortion parameter, and a sensor format size. The geometric parameter of the positioning sensor 106 or the IMU sensor 108 may include any one or more of an axial scale and an axial offset.

The system 100 includes a reference image database unit 114 , a search selection unit 116 , a matching unit 118 , a geometric transformation unit 120 , a matching optimal value estimation unit 122 , and a device geometry information estimation unit 124 . ), a 3D image information generation unit 126 , an object information extraction unit 128 , a spatial information generation unit 130 , a spatial information update unit 132 , and a spatial information database unit 134 .

The reference image database unit 114 is linked with reference geometry information including three-dimensional position data of the reference image and first external geometric data having first posture data defining a viewing direction and viewing angle set to generate the reference image. Save the reference image data. The three-dimensional position data may be two-dimensional position data, ie, XYZ coordinates obtained by adding an altitude value Z to the XY coordinates. The viewing direction and viewing angle are substantially the same as those described for the image sensor 104 . Also, the reference geometric information may further include first internal geometric data based on a geometric parameter defined in a sensor configured to generate a reference image.

The reference image is an image obtained from the outside, including at least three-dimensional position data, and is processed into reference image data that can be contrasted with image data obtained from the mobile device 102 . The sensor initially used to generate the reference image may be at least one of an image acquisition device and a laser scanning sensor (eg, Lidar) having higher positioning accuracy and image realization performance than the mobile device 102 . The reference geometric information associated with the reference image includes, for each pixel coordinate, three-dimensional position information, color information such as RGB data, first external geometric data, and first internal geometric data, and the reference image combined with the reference geometric information includes: It may be stored as reference image data.

When the sensor initially used to generate the reference image is an image sensor, the first internal geometric data is substantially the same as the image sensor 104 described in the second internal geometric data. When the first used sensor is a laser scanning sensor, the first internal geometric data may be, for example, at least one of an incident angle of each laser, a distance scale, a distance direction offset, and an axial direction offset.

When the reference image originates from a wide-area image sensor that generates a distorted image, the reference image uses flat image data generated by converting the distorted image into flat image data, and the three-dimensional position data is the three-dimensional position of the flat image data. Data is used, and the first posture data may be defined as a viewing direction and a viewing angle of the planar image data. A detailed conversion process will be described later.

When the reference image originates from the laser scanning sensor, the reference image may use virtual image data obtained by converting the point cloud data obtained from the laser scanning sensor. The 3D position data is the 3D position of the virtual image data. data, and the first posture data may be defined in a viewing direction of the virtual image data. Transformation of the point cloud data may be performed by virtual camera modeling in consideration of reflection intensity, altitude, and the like of the point cloud data. When the image sensor is additionally involved in generating the reference image, color data obtained by the image sensor when the point cloud data is converted may be included in the virtual image data.

In addition, the reference image database unit 114 opens and stores information related to a specific type of object, ie, object information, in the reference image data. For example, the object may correspond to a building that needs to be displayed on a map, a traffic sign that needs to be recognized for autonomous driving, a traffic light, a lane, and the like. In this case, the object information may include the type of the object, the overall shape, a specific geometric shape such as a line, a surface, a circle, or a sphere constituting at least a part of the object, a color, and a texture-related attribute.

On the other hand, the search selection unit 116 obtains device geometry information in relation to the image data acquired from the mobile device 102, and first and second external geometric data including three-dimensional and two-dimensional position data and posture data; A candidate group of image data that can be matched with the reference image data is searched for with reference to the first and second internal geometric data, and image data having the highest degree of matching with the reference image data is selected from the candidate group.

Here, the image data may be used not only as matrix-type data having color information, but also as a descriptor having a characteristic that can represent a specific object. In the case of a descriptor having a characteristic representing a specific object, SIFT (Scale Invariant Feature Transform) may be used as an image comparison algorithm.

In order to calculate the descriptor according to the SIFT, the order of Scale-space extrema, Key-point localization & Filtering, Orientation allocation, and descriptor generation may be performed.

Specifically, if a boundary is simply detected in the image data, a lot of meaningless edges including noise are also detected. First, the image data is blurred by applying a Gaussian filter at various levels (adjusting the σ value), When the difference between the image data and the reference image data is calculated, a characteristic robust to noise is obtained (DoG: Difference of Gaussian). When the work on the current scale is completed, the same work is performed while changing the image size to be able to respond to the scale change to obtain extrema, a robust feature at different sizes.

Even though relatively robust extrema is extracted through DoG, it still contains a lot of unnecessary information, so only meaningful information is extracted through keypoint localization and filtering. Next, a histogram according to the direction of the gradient is calculated.

In the NХN gradient window, NХNХm dimension descriptors are generated in m directions, and the selection and consistency of images are determined by comparing these descriptors.

Specifically, the search selection unit 116 searches for a candidate group of image data that can be contrasted with the reference image data based on the 3D position data and the 2D position data, and refers to the first and second posture data for a reference image among the candidate group. It is possible to select the image data having the highest degree of matching with the data.

When the image data obtained from the mobile device 102 originates from the wide-area image sensor, the search selection unit 116 performs the process of removing distortion of the wide-area image and converting it into flat image data. This can be done before the data selection process.

In the above embodiment, the search selection unit 116 selects a candidate group of image data by analyzing the first and second external geometric data including the posture data, and even the first and second internal geometric data, and image data having the highest degree of matching is described as selecting a candidate group and image data selection process based on values other than the altitude value of the 3D location data together with the 2D location data.

The matching unit 118 matches the reference image data and the selected image data based on a predetermined shape of feature geometry common between the reference image data and the selected image data, and through a geometric model such as a collinear conditional expression with reference to the feature geometry A plurality of selected image data may be combined.

A predetermined type of feature geometry may be extracted based on a specific geometric shape of an object expressed in the selected image data and the reference image data, which are estimated to be at the same location. The object may be randomly selected as an object from which feature geometries such as points, lines, planes, and columns can be easily extracted from image data, such as road signs, objects of unusual shapes, building boundaries, and the like. Even if the posture data between the selected image data and the reference image data is slightly different, the characteristic geometry adopts a predetermined shape with a high probability of being estimated as the same object, and may be, for example, a lane marked on a road surface, a road sign, a building outline, and the like.

The geometric transformation unit 120 geometrically transforms the geometry information of the device of the selected image data related to the reference image data in common feature geometries using geometric transformation modeling to include the three-dimensional position data to include the three-dimensional device. Calibrate with geometric information.

For geometric transformation modeling, one of perspective transformation, affine transformation, and homography transformation, which can reflect the geometry of image data in a matrix form, may be used, or alternative modeling may be used.

[Equation 1]

The image coordinates are two-dimensional position data of image data related to the same feature geometry, and the object coordinates are three-dimensional position data of reference image data for the same feature geometry. The camera position and posture are first external geometric data of the reference image data, and the main point position, focal length, and lens distortion are first internal geometric data.

The matching optimal value estimator 122 randomly samples the feature geometries of the image data to which the 3D device geometry information is linked, and the sampling image data in which the amount of mutual separation between the sampled feature geometries and the feature geometries of the corresponding reference data is minimized. The feature geometry of can be estimated as the matching optimal value.

Specifically, a plurality of image data sharing a feature geometry is sampled for each of the same estimated feature geometries, and reference image data having a feature geometry is linked and selected. Next, when image data and reference image data sharing the same estimated feature geometry are arranged according to the 3D device geometry information, the amount of separation between the image data and the reference image data and the feature geometries estimated to be the same is determined to be the minimum The feature geometry of the image data to be used is estimated as a matching optimal value.

The device geometry information estimator 124 may estimate the geometrically transformed 3D device geometry information in relation to the feature geometry estimated as the above-described optimal matching value as the final 3D device geometry information.

The 3D image information generating unit 126 generates 3D image information with respect to the image data of the mobile device 102 based on the final 3D device geometry information and matching information on the feature geometry.

The three-dimensional image information, like the reference image data of the reference image database unit 114, includes image data corrected by geometric transformation, three-dimensional position data linked thereto, and corrected second external posture data including corrected second posture data. It may include geometric data, final 3D device geometry information including the corrected second internal geometric data, color data, time data, and the like.

The object information extractor 128 may extract an object of a specific shape from 3D image information resulting from image data.

Specifically, the object information extractor 128 may extract object information related to the extracted object attribute based on data included in the 3D image information.

The object information extraction unit 128 detects the candidate group data related to the object identified from the image data included in the 3D image information, the final 3D device geometry information, the color data, the time data, etc. using a machine learning technique, and the detected candidate group By sequentially applying additional machine learning models to the data, the information of the object can be recognized.

Machine learning is performed in two stages: target object detection and target object recognition. In the target object detection step, candidate groups are detected through machine learning based on properties such as vectors such as shape, color, and texture of the target object. Then, an additional machine learning model is sequentially applied to the detected candidate group to recognize and classify the information indicated by the target object.

The object information extraction unit 128 may recognize various ?h objects for the surrounding environment of the mobile device included in the 3D image information, and transmit related object information to the disaster type extraction unit 132, which will be described later. The disaster type extraction unit 132 may analyze related object information by machine learning to determine whether a disaster has occurred, and in the event of a disaster, classify the disaster type by detailed location of the disaster occurrence area.

The spatial information generating unit 130, based on the object of the same reference image data as the extracted object, a predetermined coordinate system including final three-dimensional device geometry information and object information related to an object stored in the reference image data in the three-dimensional image information; For example, new spatial information according to the absolute coordinate system may be generated and transmitted to and stored in the spatial information database unit 134 . Spatial information is spatial data to be described later, and detailed information will be described later.

The disaster type extractor 132 may determine whether a disaster has occurred in the surrounding environment of the mobile device from 3D image information based on image data to the mobile device, and may classify the disaster type for each disaster occurrence region. The disaster type extraction unit 132 may classify disaster types while determining whether a disaster has occurred through machine learning on 3D image information.

The disaster information registration unit 136 may register location information and disaster types related to the disaster occurrence area in association with 3D image information. The location information on the disaster area may be disaster location information identified for each type of disaster, a shooting point of a mobile device, and the like. The disaster location information may be composed of 3D location information in which location data of the mobile device is corrected for positioning by 3D location data of reference image data. Such three-dimensional position information may be expressed in, for example, an absolute coordinate system in order to match the spatial information database unit 134 . The disaster information registration unit 136, based on the 3D location information described in the absolute coordinate system, provides the location information of the disaster occurrence area and the disaster type for each detailed location as 3D image information and spatial data (spatial information database unit 134 ) to be described later. ) can be registered in the spatial information database unit 134 in association with the spatial data of ). Accordingly, by using the calibrated location data of the mobile device, that is, the calibrated three-dimensional positioning information and the disaster information (disaster image information) obtained from the three-dimensional image information, the types of disasters classified for each location in the disaster occurrence area are displayed on the map. can express In addition, disaster information including spatial data may be displayed through the map.

In addition, the disaster information registration unit 136 may process the 3D image information by performing clustering based on the type of disaster in the disaster occurrence area and geographic proximity between the disaster occurrence areas. Data clustered in 3D image information may also be registered in the spatial information database unit 134 in association with spatial data to be described later. Accordingly, a map presented by clustering by the same or similar disaster type may be generated.

The spatial information database unit 134 structures and stores observation data previously acquired from heterogeneous sensors and data previously generated for drawing as spatial data according to a predetermined coordinate system, for example, an absolute coordinate system. Spatial data is data generated in corresponding coordinates of a predetermined coordinate system, and is observed data, internal geometry, external geometry, intrinsic data, time information, acquisition environment information, separation amount due to position error of sensor data between heterogeneous sensors, accuracy, and precision It may include related probability information, point cloud data related information, image data related information, object information, the above-mentioned disaster information and map related information. In the present embodiment, it is described that spatial data includes all of the above-described data or information, but is not limited thereto, and may be set to include only some of the above-described data/information according to design specifications.

The disaster information may include location information related to a disaster occurrence area registered through the disaster information register 136 , a disaster type for each detailed location, clustered data, and the like.

According to this embodiment, the matching between the image data having the 2D position data of the mobile device 102 and the reference image data having the 3D position data of the mobile device 102 is not a matching method between different dimensions, but a 2D image matching method. Therefore, the matching process can be performed quickly with low complexity.

In addition, 3D image information can be generated by geometrically transforming and correcting position data of image data with 3D position data with high precision and accuracy included in the reference image data, and accordingly, high precision/accuracy is required in the future. It can be usefully utilized in subsequent data processing.

In addition, in the present embodiment, the final 3D device geometry information, etc. is generated using the feature geometries with the minimum amount of mutual separation between the feature geometries that are estimated to be the same through the matching optimal value estimator 122, but the second external geometric data Since is data capable of achieving good matching between image data, the geometric transformation unit 120 may be omitted if the condition that the mutual spacing between the feature geometries is less than or equal to a threshold value is satisfied.

In addition, according to the present embodiment, image data of two-dimensional position information acquired by a low-performance image sensor or a positioning sensor of a mobile device is matched with three-dimensional image information generated from a platform composed of high-performance sensors, and geometric transformation, etc. By performing correction through , it is possible to generate reliable disaster information from high-precision 3D image information generated based on image data of a mobile device.

In addition, based on the 3D image information based on the mobile device, it is possible to accurately identify the accurate positioning information and the disaster type of the disaster-occurring area, and by clustering the disaster-occurring area by type of disaster and visualizing it on a map, etc., the disaster situation can be systematically checked can contribute to

Hereinafter, a method for generating spatial information of a mobile device using 3D image information according to another embodiment of the present invention will be described with reference to FIGS. 2 to 13 . In the present embodiment, the first and second posture data are used in the process of selecting the image data having the highest degree of matching with the reference image data, and the function of the geometric conversion unit 120 is performed. However, as described above, they may be omitted.

2 is a flowchart illustrating a disaster situation recognition method by spatial information clustering based on a mobile device image according to another embodiment of the present invention. 3 is a diagram illustrating an example of 3D image information stored in a reference database unit.

First, the search selection unit 116 obtains device geometry information including second external geometric data and second internal geometric data having two-dimensional position data and second posture data related to image data acquired from the mobile device 102 . Acquire (S205).

The second external geometric data is based on a parameter value that is changed whenever image information of the image sensor 104 is acquired due to the position and posture of the mobile device 102 in motion, that is, the position and posture of the image sensor 104 . This is the error of the observed data calculated by doing this, and in this case, it may be calculated through a predetermined formula or modeling. The second internal geometric data is an eigenvalue of the sensors 104 to 108 itself, and is an error of observation data for each sensor 104 to 108 due to a parameter maintained regardless of whether the mobile device 102 moves. The geometric parameters for each of the sensors 104 to 108 are omitted above.

Next, the search selection unit 116 is configured to perform at least three-dimensional position data and an image among reference geometry information of reference image data including first external geometric data and first internal geometric data having three-dimensional position data and first posture data. A candidate group of image data is searched based on the two-dimensional position data of the data (S210).

Here, the image data may be used not only as matrix-type data having color information, but also as a descriptor having a characteristic that can represent a specific object. In the case of a descriptor having a characteristic representing a specific object, SIFT (Scale Invariant Feature Transform) may be used as an image comparison algorithm. This will be described in detail in the description of the system 100 and will be omitted.

The reference image data is stored in the reference image database unit 114 , and the reference image data stores the above-mentioned data including 3D position data in a pixel displayed in a dark color as shown in FIG. 3 . The three-dimensional position data may be two-dimensional position data, ie, XYZ coordinates obtained by adding an altitude value Z to the XY coordinates. The first external geometric data includes the above-described first posture data together with the three-dimensional position data, and the first internal geometric data is based on a geometric parameter defined in a sensor configured to generate a reference image.

The reference image is an image obtained from the outside, including at least three-dimensional position data, and is processed into reference image data that can be contrasted with image data obtained from the mobile device 102 . The sensor initially used to generate the reference image may be at least one of an image acquisition device and a laser scanning sensor (eg, Lidar) having higher positioning accuracy and image realization performance than the mobile device 102 . The reference geometric information associated with the reference image includes, for each pixel coordinate, three-dimensional position information, color information such as RGB data, first external geometric data, and first internal geometric data, and the reference image combined with the reference geometric information includes: It may be stored as reference image data.

When the sensor initially used to generate the reference image is an image sensor, the first internal geometric data is substantially the same as the image sensor 104 described in the second internal geometric data. When a sensor initially used is a laser scanning sensor, the first internal geometric data may include, for example, any one or more of an incident angle, a distance scale, a distance direction offset, and an axial direction offset of each laser.

When the reference image originates from a wide-area image sensor that generates a distorted image, the reference image uses flat image data generated by converting the distorted image into flat image data, and the three-dimensional position data is the three-dimensional position of the flat image data. Data is used, and the first posture data may be defined as a viewing direction and a viewing angle of the planar image data.

In this regard, the process of removing distortion and generating planar image data performed in the reference image database unit 114 may be performed as described below.

When image data for a fisheye lens distorted due to a wide-area image sensor equipped with a fisheye lens is input, the reference image database 114 defines a correspondence between the distorted image data from the image data and the flat image data for reference. Each pixel can be assigned an index. Next, the reference image database unit 114 may decompose the indexed distortion image data into preset sections. Next, the reference image database unit 114 may generate the planar image data by removing the distortion of the distorted image data based on the distortion correction model of the decomposed section with reference to the planar image data for reference. The distortion correction model may be modeling based on a formula or an index of the decomposed section.

If the omnidirectional image data is input, the reference image database unit 114 may decompose the omnidirectional image data into preset sections, as described above. Next, the reference image database unit 114 may re-decompose the decomposed omnidirectional image into a preset section. Next, the reference image database unit 114 may generate flat image data by removing distortion based on a distortion correction model of the re-decomposed section, for example, an equation or an index.

This is substantially the same process even when the image data received from the mobile device 102 includes a distorted image due to the wide-area image sensor, and this process can be performed in the search selection unit 116 for inputting image data. there is.

Meanwhile, FIGS. 4 and 5 illustrate another process of removing distortion due to the optical image sensor and converting into planar image data, which is different from the process described above. 4 is a flowchart illustrating a process of converting image data acquired from a wide area image sensor into flat image data by removing distortion. 5 is a diagram schematically illustrating a process of image distortion removal and flat image data conversion.

4 and 5 , when omniazimuth image data distorted due to a wide area image sensor generating an omnidirectional image is input to the reference image database unit 114 ( S405 ), the reference image database unit 114 displays the image A virtual three-dimensional shape is restored from the distorted image data based on the camera geometry based on the distorted image data from the data (S410).

Next, the reference image database unit 114 generates plane image data by defining a virtual camera geometry and projecting it to the virtual plane camera through the geometry ( S415 ).

Alternatively, when the reference image originates from the laser scanning sensor, the reference image may use virtual image data obtained by converting point cloud data obtained from the laser scanning sensor. It is defined as 3D position data, and the first posture data may be defined as a viewing direction of the virtual image data.

Transformation of the point cloud data may be performed by virtual camera modeling in consideration of the reflection intensity and altitude value of the point cloud data located below the drawing as shown in FIG. 6 . 9 is a diagram in which the reference image database unit stores virtual image data obtained by converting point cloud data obtained from a laser scanning sensor. As a result, virtual image data may be generated as shown in the above example of FIG. 6 . When the image sensor is additionally involved in generating the reference image, color data obtained by the image sensor when the point cloud data is converted may be included in the virtual image data.

As described above, the search for the candidate group of the image data based on the 3D position data of the reference image data and the 2D position data of the image data performed by the search selection unit 116 may be performed in the same search process as the step below in FIG. 7 . there is. 7 is a diagram illustrating a process of searching for a candidate group of image data that can be matched with reference image data.

Next, the search selection unit 116 selects image data having the highest degree of matching with the reference image data from the candidate group with reference to the two-dimensional position data, the three-dimensional position data, and the first and second posture data.

Specifically, if the first and second posture data of the first and second external geometric data are different according to the viewing direction and viewing angle of the image data having the same or very similar two-dimensional position data as the reference image data, the image geometry Even the same feature geometry is recognized as different geometry, so it is difficult to match the reference image data and the image data. Accordingly, by selecting only image data similar to the first and second posture data within a predetermined range at the same position from the candidate group, the accuracy and processing speed of the subsequent matching step may be improved. In this example, a search based on direction information is performed as shown in FIG. 7 , and as shown in FIG. 8 , image data adjacent to the position and posture of the reference image data within a predetermined range is finally selected. 8 is a diagram illustrating a process of selecting image data having the highest degree of matching with reference image data.

Next, the matching unit 118 matches the reference image data and the selected image data based on a predetermined type of feature geometry common between the selected image data and the reference image data ( S220 ).

The matching unit 118 may combine a plurality of selected image data through a geometric model such as a collinear conditional expression with reference to the feature geometry.

The feature geometry of a predetermined shape may be extracted based on a specific geometric shape of the object expressed in the selected image data and the reference image data estimated to be the same location. The object may be randomly selected as an object from which feature geometries such as points, lines, planes, and columns can be easily extracted from image data, such as road signs, objects of unusual shapes, building boundaries, and the like. Even if the posture data between the selected image data and the reference image data is slightly different, the characteristic geometry adopts a predetermined shape with a high probability of being estimated as the same object, and may be, for example, a lane marked on a road surface, a road sign, a building outline, and the like.

In the common feature geometries, the geometric transformation unit 120 geometrically transforms the geometry information of the device of the selected image data for the reference image data by geometric transformation modeling to include the three-dimensional position data, and corrects it to the three-dimensional device geometry information. do.

For the geometric transformation modeling, one of perspective transformation, affine transformation, and homography transformation that can reflect the geometry of image data in a matrix form may be used, or alternative modeling may be used.

9 is a diagram illustrating a process of calculating two-dimensional position data of image data and three-dimensional position data of reference image data corresponding to the two-dimensional position data of image data. In the case of using the collinear conditional expression, image data is geometrically transformed into three-dimensional position coordinates of the reference image data and corrected in the same feature geometry. 10 is a diagram illustrating an example in which a position and a viewing direction of image data of a mobile device are corrected with final 3D device geometry information. The dotted line is the two-dimensional position data of the image data obtained from the mobile device 102, and the solid line is the mobile device ( 102), indicating that it is converted to a higher precision than that of the positioning sensor 106.

Next, the matching optimal value estimator 122 randomly samples the feature geometries of the image data to which the 3D device geometry information is linked, so that the amount of mutual separation between the sampled feature geometries and the feature geometries of the corresponding reference data is minimized. The feature geometry of the sampled image data is estimated as a matching optimal value (S230). The detailed process of the matching optimal value will be omitted as described above.

Next, the device geometry information estimator 124 estimates the geometrically transformed 3D device geometry information in relation to the feature geometry estimated as the above-described optimal matching value as the final 3D device geometry information (S235).

Next, the 3D image information generating unit 126 generates 3D image information with respect to the image data of the mobile device 102 based on the final 3D device geometry information and matching information on the feature geometry ( S240 ). .

The three-dimensional image information, like the reference image data of the reference image database unit 114, includes image data corrected by geometric transformation, three-dimensional position data linked thereto, and corrected second external posture data including corrected second posture data. It may include geometric data, final 3D device geometry information including the corrected second internal geometric data, color data, time data, and the like.

Next, the disaster type extraction unit 132 determines whether a disaster has occurred in the surrounding environment of the mobile device from 3D image information based on image data to the mobile device, and classifies the disaster type for each disaster occurrence region (S245) ).

The disaster type extraction unit 132 may classify disaster types while determining whether a disaster has occurred through machine learning on 3D image information.

The object information extraction unit 128 may recognize various ?h objects for the surrounding environment of the mobile device included in the 3D image information, and transmit related object information to the disaster type extraction unit 132, which will be described later.

A process by which the object information extraction unit 128 recognizes an object from 3D image information is as follows.

The object information extractor 128 may extract an object of a specific shape from 3D image information resulting from image data. Specifically, the object information extractor 128 may extract object information related to the extracted object attribute based on data included in the 3D image information.

The object information extraction unit 128 detects the candidate group data related to the object identified from the image data included in the 3D image information, the final 3D device geometry information, the color data, the time data, etc. using a machine learning technique, and the detected candidate group By sequentially applying additional machine learning models to the data, the information of the object can be recognized.

Next, the spatial information generator 130 includes, in the 3D image information, the final 3D device geometry information and object information related to the object stored in the reference image data, based on the object of the same reference image data as the extracted object. New spatial information can be generated according to a predetermined coordinate system, for example, an absolute coordinate system.

The spatial information generating unit 130 may transmit and store new spatial information to the spatial information database unit 134 . Spatial information is data generated in corresponding coordinates of a predetermined coordinate system, and is observed data, internal geometry, external geometry, intrinsic data, time information, acquisition environment information, separation amount due to position error of sensor data between heterogeneous sensors, accuracy, and precision It may include related probability information, point cloud data related information, image data related information, object information, disaster information, and map related information.

The final three-dimensional device geometry information and object information are included in the three-dimensional image information to generate spatial information, and based on this, the map on the right can be produced through a drawing process.

The disaster type extraction unit 132 may analyze related object information by machine learning to determine whether a disaster has occurred, and in the event of a disaster, classify the disaster type by detailed location of the disaster occurrence area.

11 is a diagram illustrating a process of machine learning for classifying by disaster type.

The disaster type extraction unit 132 may machine learning a disaster recognition/type determination model by using various types of disasters for a predetermined period as learning data, either alone or in connection with the object information extraction unit 128 . The disaster type extractor 132 may determine whether at least one of various disaster types is included in the disaster image using the disaster recognition/type determination model. For example, the disaster type may be a landslide, a restoration work, a structure due to wind and flood damage, pollution due to soil, soil, etc. as shown in FIG. 11 . The disaster type extraction unit 132 may transmit the determination result to the disaster information registration unit 136 . For example, the determination result may include whether a disaster has occurred in an object included in an image of the surrounding environment, a disaster type, and a probability value for the determined disaster type.

Next, the disaster information registration unit 136 registers the location information and the disaster type related to the disaster occurrence area in association with the 3D image information (S250).

The location information on the disaster area may be disaster location information identified for each type of disaster, a shooting point of a mobile device, and the like. The disaster location information may be composed of 3D location information in which location data of the mobile device is corrected for positioning by 3D location data of reference image data. Such three-dimensional position information may be expressed in, for example, an absolute coordinate system in order to match the spatial information database unit 134 . The disaster information registration unit 136, based on the 3D location information described in the absolute coordinate system, provides the location information of the disaster occurrence area and the disaster type for each detailed location as 3D image information and spatial data (spatial information database unit 134 ) to be described later. ) can be registered in the spatial information database unit 134 in association with the spatial data of ). Accordingly, using the calibrated location data of the mobile device, that is, the calibrated three-dimensional positioning information and the disaster information (disaster image information) acquired from the three-dimensional image information, as shown in FIG. 12, the Disaster types can be displayed on the map. In addition, disaster information including spatial data may be displayed through the map. 12 is a diagram illustrating an example of labeling a disaster type for each disaster location in a disaster occurrence area.

Next, the disaster information registration unit 136 processes the 3D image information by performing clustering based on the type of disaster in the disaster occurrence area and geographic proximity between the disaster occurrence areas (S255).

Data clustered in 3D image information may also be registered in the spatial information database unit 134 in association with spatial data to be described later. The disaster information registered in the spatial information database unit 134 may include location information related to a disaster occurrence area, a disaster type for each detailed location, clustered data, and the like. Accordingly, as shown in FIG. 13 , a map presented by clustering according to the same and similar disaster types may be generated. 13 is a diagram illustrating an example of performing clustering for each disaster type in a disaster occurrence area.

In addition, according to the present embodiment, image data of two-dimensional position information acquired by a low-performance image sensor or a positioning sensor of a mobile device is matched with three-dimensional image information generated from a platform composed of high-performance sensors, and geometric transformation, etc. By performing correction through , it is possible to generate reliable disaster information from high-precision 3D image information generated based on image data of a mobile device.

In addition, based on the 3D image information based on the mobile device, it is possible to accurately identify the accurate positioning information and the disaster type of the disaster-occurring area, and by clustering the disaster-occurring area by type of disaster and visualizing it on a map, etc., the disaster situation can be systematically checked can contribute to

The components constituting the apparatus shown in FIG. 1 or the steps according to the embodiments shown in FIG. 2 may be recorded in a computer-readable recording medium in the form of a program realizing the function. Here, the computer-readable recording medium refers to a computer-readable recording medium in which information such as data or a program is stored by electrical, magnetic, optical, mechanical, or chemical action. Among these recording media, for example, portable storage, flexible disk, magneto-optical disk, CD-ROM, CD-R/W, DVD, DAT, memory card, etc. are available as those that can be separated from the computer. In addition, as a recording medium fixed to a mobile device and a computer, there is a solid state disk (SSD), a hard disk, a ROM, and the like.

Example methods of the present disclosure are expressed as a series of operations for clarity of description, but this is not intended to limit the order in which the steps are performed, and if necessary, each step may be performed simultaneously or in a different order. In order to implement the method according to the present disclosure, other steps may be included in addition to the illustrated steps, other steps may be excluded from some steps, or additional other steps may be included except some steps.

Various embodiments of the present disclosure do not list all possible combinations, but are intended to describe representative aspects of the present disclosure, and matters described in various embodiments may be applied independently or in combination of two or more.

In addition, the method according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed by various computer devices and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For implementation by hardware, one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose It may be implemented by a processor (general processor), a controller, a microcontroller, a microprocessor, and the like.

The scope of the present disclosure is software or machine-executable instructions (eg, operating system, application, firmware, program, etc.) that cause operation according to the method of various embodiments to be executed on a device or computer device, and such software or a non-transitory computer-readable medium in which instructions and the like are stored and executed on an apparatus or computer.

Claims (10)

a reference image database unit for storing reference image data in association with three-dimensional position data and reference geometry information including first external geometry data having first posture data defining a viewing direction set to generate a reference image;
Obtaining device geometry information including second external geometric data having at least two-dimensional position data in relation to the image data acquired from the mobile device, and referring to the two-dimensional position data and the three-dimensional position data, the reference image data and a search selection unit that searches for a candidate group of the image data that can be matched, and selects the image data having the highest degree of matching with the reference image data from among the candidate group;
a matching unit configured to match the reference image data and the selected image data based on a predetermined shape of feature geometries that are common between the reference image data and the selected image data;
In common feature geometries, the geometric information of the device of the selected image data matched with the reference image data is geometrically transformed by geometric transformation modeling so as to include the three-dimensional position data to a three-dimensional device. a geometrical conversion unit that corrects the geometrical information;
a 3D image information generator configured to generate 3D image information with respect to the image data of the mobile device based on the 3D device geometry information and the matching information for the feature geometry;
a disaster type extraction unit for determining whether a disaster has occurred in the surrounding environment of the mobile device from the 3D image information based on the image data to the mobile device, and classifying disaster types for each disaster occurrence region; and
A disaster situation recognition system by spatial information clustering based on a mobile device image, comprising a disaster information registration unit that registers the location information related to the disaster area and the disaster type in association with the 3D image information.
The method of claim 1,
The disaster type extraction unit determines whether the disaster has occurred through machine learning on the 3D image information and classifies the disaster type.
The method of claim 1,
The disaster information registration unit processes the 3D image information by performing clustering based on a disaster type of the disaster occurrence area and geographic proximity between the disaster occurrence areas.
The method of claim 1,
The second external geometric data further includes second posture data defining a viewing direction set to generate the image,
The search selection unit searches for a candidate group of the image data that can be contrasted with the reference image data based on the 3D position data and the 2D position data, and refers to the first and second posture data of the candidate group. A disaster situation recognition system that selects the image data having the highest degree of matching with the image data.
5. The method of claim 4,
The reference geometric information further includes first internal geometric data based on a geometric parameter defined in a sensor configured to generate the reference image, wherein the device geometric information is based on a geometric parameter defined in an image sensor configured to generate the image Further comprising second internal geometric data, the selection of the image data having the highest degree of matching in the search selection unit and the correction to the 3D device geometry information in the geometric transformation unit are performed in the first and second A disaster situation awareness system, further including referenced internal geometric data.
The method of claim 1,
When the reference image originates from the laser scanning sensor, the reference image uses virtual image data obtained by converting the point cloud data obtained from the laser scanning sensor, and the 3D position data is the virtual image data is defined as three-dimensional position data of, and the first posture data is defined as a viewing direction of the virtual image data.
The method of claim 1,
By randomly sampling the feature geometries of the image data to which the 3D device geometry information is linked, the sampled feature geometries and the corresponding feature geometries of the reference image data are matched with the feature geometries of the sampled image data in which the mutual separation amount is minimized a matched optimal value estimator for estimating an optimal value; and
Further comprising a device geometry information estimator for estimating the geometrically transformed 3D device geometry information as final 3D device geometry information in relation to the feature geometry estimated as the matching optimal value,
The 3D device geometric information among the information based on the generation of the 3D image information uses the final 3D device geometric information, a disaster situation recognition system.
A viewing direction set to obtain device geometry information including second external geometric data having at least two-dimensional position data in relation to image data acquired from a mobile device, and to generate three-dimensional position data and a reference image linked to the reference image data A candidate group of the image data that can be matched with the reference image data is searched for by referring to the three-dimensional position data and the two-dimensional position data among reference geometric information including first external geometric data defining a search selection step of selecting the image data having the highest degree of matching with the image data;
a matching step of matching the reference image data and the selected image data based on common features of predetermined shapes between the reference image data and the selected image data;
Geometric transformation for correcting the geometric information of the device into three-dimensional device geometric information by geometric transformation by geometric transformation modeling so that the geometric information of the device of the selected image data matched with the reference image data includes the three-dimensional position data step;
a 3D image information generating step of generating 3D image information with respect to the image data of the mobile device based on the 3D device geometry information and matching information for the feature geometry;
a disaster type extraction step of determining whether a disaster has occurred in the surrounding environment of the mobile device from the 3D image information based on the image data to the mobile device, and classifying the disaster type for each disaster occurrence region; and
A disaster situation recognition method by spatial information clustering based on a mobile device image, comprising a disaster information registration step of registering the location information related to the disaster area and the disaster type in association with the 3D image information.
9. The method of claim 8,
A disaster situation recognition method, wherein the disaster type extraction unit determines whether the disaster occurs through machine learning on the 3D image information and classifies the disaster type.
9. The method of claim 8,
The disaster information registration step is a disaster situation recognition method of processing the 3D image information by performing clustering based on a disaster type of the disaster occurrence area and geographic proximity between the disaster occurrence areas.

Images