Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T17/00—Three-dimensional [3D] modelling for computer graphics
  • G06T17/05—Geographic models
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
  • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
  • G01C21/20—Instruments for performing navigational calculations
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
  • G01S17/06—Systems determining position data of a target
  • G01S17/08—Systems determining position data of a target for measuring distance only
  • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/88—Lidar systems specially adapted for specific applications
  • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
  • G01S5/16—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using electromagnetic waves other than radio waves
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
  • G01S5/16—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using electromagnetic waves other than radio waves
  • G01S5/163—Determination of attitude
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/24—Classification techniques
  • G06F18/243—Classification techniques relating to the number of classes
  • G06F18/2431—Multiple classes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F18/00—Pattern recognition
  • G06F18/20—Analysing
  • G06F18/25—Fusion techniques
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/10—Terrestrial scenes
  • G06V20/176—Urban or other man-made structures
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/10—Terrestrial scenes
  • G06V20/188—Vegetation
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2210/00—Indexing scheme for image generation or computer graphics
  • G06T2210/56—Particle system, point based geometry or rendering
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/10—Terrestrial scenes
  • G06V20/194—Terrestrial scenes using hyperspectral data, i.e. more or other wavelengths than RGB

Definitions

• the present disclosure relates to a system and method for facilitating the generation of geographical information.
• 3D sensor three-dimensional type sensor
• HD High Definition
• LiDAR Light Detection and Ranging
• the LiDAR sensor measures distance to an object by illuminating the object with pulsed laser light and detecting the reflected pulses, and image data is obtained using the detected distance.
• processing of image data will vary depending on the types of the geographical information that the user desires to generate. Depending on the type of the geographical information, the user is required to consider necessary geographical factors and process the image data in an appropriate way.
• the invention or disclosure seeks to provide a system and method to reduce the user’s manual and laborious work in generation of geographical information.
• the technical solution is provided in the form of a system and method for facilitating generation of geographical information.
• the system comprises a processor operable to analyse image data obtained by an imaging module. Thereafter, the processor is operable to extract at least one feature from the image data based on the analysis. The extracted feature is associated with the geographical information that the user desires to generate. The processor is then operable to generate the geographical information using the extracted feature.
• the processor is operable to generate the geographical information that the user desires to generate, in real-time or near real-time, without the user’s manual and laborious work.
• a system for facilitating generation of geographical information comprising: an imaging module operable to obtain image data; and a processor operable to receive the image data and generate a point cloud having a plurality of data points using the image data, wherein the processor is further operable to analyse the point cloud to extract at least one feature associated with the geographical information, and generate the geographical information using the extracted feature.
• the imaging module comprises at least one 3D sensor operable to generate 3D image data as the image data.
• the generation of the point cloud includes georeferencing of the image data.
• system further comprises a position and orientation system (POS) operable to send position and/or orientation related data to the processor.
• POS position and orientation system
• the processor is operable to parse the sent data and use the parsed information to perform the georeferencing of the image data. In some embodiments, the processor is operable to correct the point cloud radiometrically and/or geometrically.
• the processor is operable to compute an octree, and divide the point cloud into a plurality of cells based on the computed octree so that each of the plurality of cells have a same size.
• the processor is operable to compute a normal for each of the data points and create an eigenvalue for each of the plurality of cells based on the computed normal.
• the processor is operable to compute at least one geometrical attribute for the each of the plurality of cells by normalizing the eigenvalues of the each of the plurality of cells.
• the processor is operable to segment the point cloud according to the geometrical attribute. In some embodiments, the processor is operable to classify the each of the data points, based on the segmented point cloud according to the geometrical attribute.
• the imaging module further comprises at least one 2D sensor operable to generate 2D image data.
• the processor is operable to merge the 2D image data and the 3D image data.
• the processor is operable to re-compute the at least one geometrical attribute for the each of the plurality of cells and/or re-classify the each of the data points, based on the merged image data. In some embodiments, the processor is operable to receive a selection of a type of the geographical information to be generated and extract the at least one feature associated with the geographical information to be generated.
• the geographical information includes at least one of a high definition map, plant information or public infrastructure information.
• the feature includes at least one of geographical feature or object feature.
• the imaging module is operable to provide the image data which was obtained previously to the processor.
• a method for facilitating generation of geographical information comprising: obtaining image data at an imaging module; receiving, at a processor, the image data from the imaging module; generating, at the processor, a point cloud having a plurality of data points using the image data; analysing, at the processor, the point cloud; extracting, at least one feature associated with the geographical information; and generating, at the processor, the geographical information using the extracted feature.
• Fig. 1 shows a block diagram in accordance with some embodiments of the present invention.
• Fig. 2 shows a flow diagram in accordance with some embodiments of the present invention.
• Fig. 3 shows another block diagram in accordance with some embodiments of the present disclosure.
• Fig. 4 shows another flow diagram in accordance with some embodiments of the present disclosure.
• Fig. 5 shows a flow diagram regarding a generation of geographical information in accordance with some embodiments of the present disclosure.
• FIG. 1 shows a block diagram in accordance with some embodiments of the present disclosure.
• a system 100 may comprise an imaging module 1 10 and a processor 120.
• the system 100 may further comprise a Position and Orientation System (POS) module 130.
• POS Position and Orientation System
• the imaging module 1 10 may comprise at least one 3D sensor 1 1 1 and/or at least one 2D sensor 1 12.
• the 3D sensor 1 1 1 may comprise, but not be limited to, a Light Detection and Ranging (LiDAR) sensor (also referred to as“LiDAR scanner”).
• the 2D sensor 1 12 may comprise, but not be limited to, an RGB camera, a multispectral imager and/or a hyperspectral imager.
• the 3D sensor 1 1 1 is operable to generate 3D image data
• the 2D sensor 1 12 is operable to generate 2D image data.
• the imaging module 1 10 may generate image data by capturing image(s) of an object.
• the LiDAR sensor detects distance to an object by illuminating the object with pulsed laser light and detecting the reflected pulses, and generates the image data using the detected distance.
• the image data may include, but not be limited to, at least one of raw data which is generated by capturing the image(s) of the object and/or data processed from the raw data (referred to as“processed data”).
• the imaging module 1 10 is operable to provide the image data to the processor 120 in real-time or near real-time, for rapid assessment and time critical decision making by the processor 120.
• the imaging module 1 10 can obtain the image data from an external database and/or a server.
• the imaging module 1 10 may receive the image data from the external database and/or the server, and then provide the image data to the processor 120.
• the imaging module 110 can provide the image data which was generated previously to the processor 120.
• the imaging module 1 10 may have an internal database.
• the internal database may include one or more memory units for storing the image data which was generated previously.
• the imaging module 1 10 may extract the image data from the internal database and then provide the image data to the processor 120.
• the POS 130 may comprise an Inertial Navigation System (INS) module 131 and a Global Navigation Satellite System (GNSS) module 132.
• INS Inertial Navigation System
• GNSS Global Navigation Satellite System
• the POS 130 may be operable to generate orientation and/or position related data through module 131 and/or module 132.
• the POS 130 may transmit the position and/or orientation related data to the processor 120.
• the position and/or orientation related data is sent to the processor 120 in real-time or near real-time.
• the sending of the position and/or orientation related data may be via streaming.
• the processor 120 may include an Explicit Data Graph Execution (EDGE) processor.
• the processor 120 can be installed on a portable device, such as a backpack, mobile and/or airborne platform.
• the portable device may include a mounting structure, such as a frame, for items to be affixed or attached thereon/thereto.
• the processor 120 may include various form factors based on a platform that the processor 120 is used on.
• the processor 120 may include at least one GPU card.
• the processor 120 may include at least one Graphics Processing Unit (GPU).
• the GPU may be integrated in the processor 120 in the form of a card as a hardware component with dedicated software installed in the processor 120.
• the processor 120 may have a casing that fulfils certain requirements, e.g., to be operable under outdoor conditions.
• the casing may be waterproof, dustproof and/or vibration proof.
• the casing meets the requirements of IP68 (also referred to as IP Code, International Protection Marking, IEC standard 60529 published by the International Electrotechnical Commission (IEC), or European standard EN 60529).
• the processor 120 can be an off the shelf solution, having a small footprint case like that known under the commercial name ROSIE from the company Connecttech (http://connecttech.com/product/rosie-embedded-system-with-nvidia- jetson-tx2-tx1/), a rugged GPGU Computing Server as available from the company Mercury system (https://www.mrcv.com/ruqqed-servers/qpqpu-computinq-servers/), or it may comprise a custom casing, e.g., milled out of plastic or aluminium using a CNC machine or a print out with a 3D printer.
• ROSIE commercial name
• Connecttech http://connecttech.com/product/rosie-embedded-system-with-nvidia- jetson-tx2-tx1/
• a rugged GPGU Computing Server as available from the company Mercury system (https://www.mrcv.com/ruqqed-servers/qpqpu-
• the processor 120 is operable to synchronize and control the imaging module 1 10 and/or the POS 130.
• the processor 120 is also operable to process and analyse the image data in real-time or near real-time.
• the GPU is operable to process the image data for output to a display device.
• the processor 120 is further operable to generate geographical information using the processed and analysed image data in real-time or near real-time, thus facilitating real- time generation of data as compared to conventional ways. Wit In some embodiments, the processor 120 extracts at least one necessary feature which is associated with the geographical information that a user desires to generate, from the processed and analysed image data. Thereafter, the processor 120 generates the geographical information using the extracted factor(s) or information. This process can be achieved automatically, without the user’s manual and laborious work. It may be appreciated that the imaging module 1 10, the processor 120 and the POS 130 are linked together via a communication network, for example a Local Area Network (LAN) such as Ethernet or a wireless communication network which can include Wi-Fi, Bluetooth, or other mobile wireless networks. The system 100 can achieve real-time mapping and monitoring for a range of mobile platforms via the communication network.
• LAN Local Area Network
• Bluetooth wireless communication network
• the system 100 provides a General-Purpose computing on Graphics Processing Units (GPGPU) hardware-software solution combining the imaging module 1 10 and the POS 130.
• the system 100 is able to collect the image data and position and/or orientation related data, and process the image data including georeferencing and analysis.
• the analysis results may be streamed to the user, for example a distant viewer, via the LAN in real-time and near real-time, for immediate inspection.
• the system 100 is able to generate geographical information using the image data, in real-time or near real-time, without any needs for the user to perform post-processing tasks.
• the generated geographical information may be streamed to the user, for example a distant viewer, via the LAN in real-time and near real-time, for immediate inspection.
• Fig. 2 shows a flow diagram in accordance with some embodiments of the present disclosure.
• the system 100 may comprise the imaging module 110 and the processor 120.
• the system 100 may further comprise the POS 130.
• the imaging module 1 10 obtains the image data (S210).
• the imaging module 1 10 may comprise at least one 3D sensor 1 1 1.
• the 3D sensor 1 11 generates 3D image data, for example LiDAR data, as the image data.
• the processor 120 then receives the image data from the imaging module 1 10 (S220).
• the processor 120 may receive or collect the 3D image data from the 3D sensor 1 1 1 in real-time or near real-time, via a communication network, for example a LAN.
• the processor 120 After receiving the 3D image data from the 3D sensor 1 11 , the processor 120 performs a georeferencing of the 3D image data.
• the georeferencing is performed to generate one or more point cloud. It may be appreciated that the position and/or orientation related data generated from the POS 130 can be used to perform the georeferencing of the 3D image data.
• the POS 130 sends the position and/or orientation related data to the processor 120 in real-time or near real-time. For example, the POS 130 streams a string of binary packets through a User Datagram Protocol (UDP) network connection.
• UDP User Datagram Protocol
• the UDP network connection allows the POS 130 to act as a server with respect to the processor 120 and to continuously stream the position and/or orientation related data through the
• the processor 120 receives the position and/or orientation related data, for example the string of binary packets, from the POS 130 and parses the string of binary packets to find needed information.
• the packets may contain, but not be limited to, position, velocity, attitude, speed and dynamics related data reported in a reference frame, for example POS 130 reference frame. It may be appreciated that these data can be a variable value that are fed into an equation, for example a LiDAR equation, to perform the georeferencing.
• the processor 120 may use the parsed information to perform the georeferencing of the image data.
• coordinates in a sensor frame are transformed to coordinates in a mapping frame, for example ECEF (earth-centred, earth-fixed) coordinates.
• ECEF earth-centred, earth-fixed
• the processor 120 then generates the point cloud using the georeferenced image data (S230).
• the point cloud includes a set of data points.
• the processor 120 may correct the point cloud radiometrically and/or geometrically.
• an intensity value of the point cloud may be corrected geometrically and calibrated radiometrically.
• recorded intensity may suffer from some deformations due to geometry of measurements and environment effect. Therefore a radiometric and geometric correction to the intensity value may be applied before being able to use it.
• the recorded intensity may be a black image and white image, where the black image may indicate low intensity and the white image may indicate high intensity.
• the processor 120 analyses the point cloud (S240).
• the analysis may include, but not be limited to one or more of the following list:- computation of a tree data structure, such as an octree, computation of normal, computation of a geometrical attribute, segmentation of the point cloud and classification of each of the data points.
• the processor 120 may compute the octree and divide the point cloud into a plurality of cells based on the computed octree, so that each of the plurality of cells have the same size, regardless of the number of points included in each cell.
• each cell may be in the form of a cube. Due to the cells having the same size, spatial searches and subsequent processing can be expedited.
• the processor 120 may compute the normal for each of the data points, and create eigenvalues and/or eigenvectors for each of the plurality of cells based on the computed normal.
• the normal may be a vector perpendicular of the best-fitting plane for the set of the data points.
• the eigenvalues and/or eigenvectors for each of the plurality of cells are for a covariance matrix created from each of the data points’ nearest neighbours.
• the processor 120 may normalize the eigenvalues of each of the plurality of cells and compute at least one geometrical attribute for each of the plurality of cells.
• the at least one geometrical attribute may include, but not be limited to, linearity, planarity, scattering, omnivariance, anisotropy, recitropy, local curvature, normal values, intensity, elevation and delta elevation with regards to an Above Ground Elevation (AGE).
• AGE Above Ground Elevation
• the processor 120 may then segment the point cloud according to the geometrical attribute. It may be appreciated that at least one of the geometrical attributes outlined above may be used to segment the point cloud, in particular elevation, normal and/or intensity, homogeneity/variation and/or (eigen) value.
• the processor 120 then classifies each of the data points, based on the segmented point cloud according to the geometrical attribute.
• the data points in each cell are classified according a predetermined set of classes, for example ASPRS (American Society for Photogrammetry and Remote Sensing) LiDAR classes. Thereafter, the data points may be meshed and used as level zero for the AGE estimation. Thus, each of the data points can have a class and the AEG elevation value.
• the processor 120 extracts at least one feature associated with the geographical information (S250).
• the processor 120 receives a selection of a type of the geographical information to be generated from the user.
• the geographical information may include, but not limited to, High Definition maps resulting from 3D feature extraction, classification and/or segmentation, road and pavement surface condition monitoring and/or maintenance, change detection in natural and/or build environments, or plant information such as tree health.
• the processor 120 can suggest various types of geographical information, and the user can select at least one geographical information that the user desires to generate.
• the processor 120 may output, to a display, information with regard to the various types of geographical information so that the user can select at least one geographical information.
• the processor 120 extracts at least one feature associated with the selected geographical information.
• the feature may include, but not be limited to, at least one of geographical feature or object feature.
• NDVI Normalized Difference Vegetation Index
• the features may be extracted and/or determined.
• Low values of NDVI for example 0.1 and below, may indicate to non-vegetated areas.
• Moderate values of NDVI for example 0.2 to 0.3, may indicate shrub and grass.
• High values of NDVI for example 0.6 to 0.8, may indicate trees and forests.
• the processor 120 generates the geographical information using the extracted feature (S260).
• the imaging module 1 10 further comprises at least one 2D sensor 1 12. Therefore, a combination of the 3D sensor 1 11 and the 2D sensor 1 12 can be used.
• the 2D sensor is operable to generate 2D image data, for example 2D images.
• the 3D sensor 1 1 1 and the 2D sensor 1 12 may obtain the 3D image data and the 2D image data respectively, with respect to the same object or background.
• the processor 120 may receive or collect the 3D image data from the 3D sensor 1 1 1 and the 2D image data from the 2D sensor 1 12 in real-time or near real-time, via the communication network.
• the processor 120 may match the 3D image data and the 2D image data based on the object or background containing in the 3D image data and the 2D image data. After receiving the 3D image data and the 2D image data from the 3D sensor 11 1 and the 2D sensor 1 12 respectively, the processor 120 performs a georeferencing of the 3D image data as well as the 2D image data, in real-time or near real-time.
• the processor 120 then generates the point cloud using the georeferenced image data.
• the processor 120 may correct the point cloud radiometrically and/or geometrically.
• some indices such as NDVI, SR (Simple Ratio Vegetation Index) and PRI (Photochemical Reflectance Index) are extracted or generated from bands ratio. The indices are used to analyse the point cloud.
• the processor 120 may merge and/or fuse the 2D image data and the 3D image data.
• the processor 120 can re-compute the at least one geometrical attribute for each of the plurality of cells and/or re-classify each of the data points, based on the merged image data.
• the processor 120 can select and perform either re-computation of the geometrical attribute or re-classification of each of the data points.
• the processor 120 can add other geometrical attributes, for example vegetation health, chlorophyll concentration, etc. Addition of the geometrical attributes to the data points is limited to first return points as the 2D image data come from 2D sensor 112 which is a passive sensor and does not have penetrating capability in term of vision.
• the 2D image data of vegetation index may be used to re-enforce and affine the vegetation delineation in the point cloud.
• the fusion of the 2D image data and the 3D image data may rely on photogrammetric and ray tracing principles.
• the processor 120 performs further process and analysis of the enhanced point cloud by 2D and 3D metrics extraction such as Digital Surface Model (DSM), Digital Terrain Model (DTM), isolines, breaklines, points of interest, etc., in real- time or near real-time.
• the results may be stored in LAS or LAZ format for point cloud metrics and in GeoTIFF format for raster metrics.
• the point cloud which includes the actual 3D image data may be stored as binary file format such as LAS or LAZ format. This is for the raw data and the processed data.
• the processor 120 may produce three (3) types of data from the point cloud.
• the types of data are as follows: point cloud for classification and segmentation; raster, for example 2D images, for the DSM, DTM, etc.; and/or vector for isolines and breaklines.
• Fig. 3 shows a block diagram in accordance with some embodiments of the present disclosure.
• Fig. 3 shows a synthetic layout of a relationship between a server 121 and a client 122 of the processor 120, for example an edge processor.
• the server 121 of the processor 120 directly interfaces with the 3D sensor 1 1 1 , the 2D sensor 1 12 and/or the POS 130.
• the processor 120 comprises one server 121 and one client 122.
• at least part of the processor 120 may be related to at least one process for the server 121 and at least part of the processor 120 may be related to at least one process for the client 122.
• additional client for example one or more GPUs, can be added to the processor 120.
• an integrated device can include the processor 120, the 3D sensor 11 1 , the 2D sensor 1 12 and/or the POS 130.
• the system 100 is able to perform an efficient parallelization of the different processes that take place either in a synchronous and asynchronous way.
• the system 100 also allows the processor 120 to be scaled up or down based on the system’s 100 physical setup. In this manner, the processor 120 can adjust the data volume to be processed in real-time or near real-time.
• the server 121 can directly interface with the 3D sensor 1 1 1 and the 2D sensor 1 12 via a Transmission Control Protocol/Internet Protocol (TCP/IP) network connection.
• a stream Reader3D module 123a and a streamReader2D module 123b can retrieve 3D image data streams and 2D image data streams respectively.
• the 3D sensor 1 1 1 is programmatically accessible through a predetermined library which communicates with system 100 via the TCP/IP connection.
• the library may be a documented platform independent software library for control of and data retrieval from the 3D sensor 1 1 1.
• the library is operable to configure and control the 3D sensor 1 1 1 and to retrieve and parse the measurements from the 3D sensor 1 1 1. It may be appreciated that the system 100 is compatible with any scanner (e.g. V-Line scanner) as the 3D sensor 1 1 1 , whether the scanner is an airborne, mobile or terrestrial scanner.
• the predetermined library should be compatible to the hardware of the sensor/scanner.
• the server 121 can directly interface with the POS 130 via a User Datagram Protocol (UDP) network connection.
• UDP User Datagram Protocol
• a streamReaderPOS module 123c can retrieve position and/or orientation related data streams.
• each data packet of a data stream may have an individual timestamp, as shown in 310a, 320a and 330a of Fig. 4.
• the timestamp is used to cross- reference each data packet to perform a georegistration on the GPU of the server 121 using the dataGeorefHandler module 124.
• the dataGeorefHandler module 124 may be an implementation of photogrammetric and LiDAR equations applied to the 2D image data and 3D image data.
• the raster metrics and point clouds are stored locally on a server-internal hard drive.
• the server 121 may send a series of messages to one or more clients 122 to inform that raw data sets, for example the raster metrics and point clouds, are available for further processing.
• each client 122 may ask the server 121 to provide required data to perform their own processing tasks by metricXHandler 125a-125n.
• the required data may include, but not be limited to, specific metrics such as Digital Surface Models (DSM), Digital Terrain Models (DTM), etc.
• the client 122 may copy the results to the server-internal hard drive in a folder that is accessible through a Samba server 126 (file service on the LAN).
• a system installation file including system installation parameters can be required by the dataGeorefHandler module 124.
• the system installation parameters are unique to a system physical installation, as it concerns the actual spatial distribution and orientation of the 3D sensor 11 1 and the 2D sensor 112 with respect to the POS 130 and to a GPS antenna.
• the various parameters can be entered into a predefined XML file that can be stored in the server 121. Therefore, a simple text editor can be required to configure the processor 120.
• Fig. 4 shows a flow diagram in accordance with some embodiments of the present disclosure. Details of elements 310a, 320a, 330a, 340a, 350a, 360a, 360b, 360c, 370a, 370b, 390a, 390b, 390c, and steps S320 to S390 will be described with reference to figure 4.
• the streamReader module 123 is connected to the POS module 130, the 3D sensor(s) 1 1 1 and 2D sensor(s) 1 12, respectively and to the system configuration module (system installation module) 140.
• the POS module 130 sends to the streamReader module 123, position and attitude information at S330a.
• the 3D sensor 1 1 1 sends to 123 spatial coordinates of target information with reference to the 3D imaging module 1 1 1. It may also send associated attributes information, e.g. of a target, such as colour, reflectance.
• the 2D module 1 12 sends to the streamReader module 123 imaging information in the form of a 2D vector with a number of associated layers of information where each layer could have information such has Red, Green or Blue colour information, or reflectance measurements in various spectral bands.
• the system installation module 140 will pass to the streamReader module 123 the information contained in an XML file. These information are related to the physical installation of each sensors in relation to each other, e.g., including the vehicle on which the system is mounted. Data streams 310a, 320a and 330a have one field of information in common, the timestamp which actually comes from the synchronization of the 3D sensor 1 1 1 and the 2D sensor 1 12 by the POS module’s Pulse Per Second (PPS) signal. Streams 310a, 320a and 330a are synchronized by the streamReader module 123 as each stream will be receive with its own rate.
• PPS Pulse Per Second
• the information is pass to the dataGeorefHandler module 124 where the actual point cloud (360a) is georeferenced for data coming out of module 1 1 1 and images are generated and georeferenced (360c) from data coming out of module 1 12.
• the dataGeorefHandler module 123 might also generate a Total Propagation Uncertainty (360b) information for each point of the point cloud.
• Point cloud 360a and associated raster images 360c are pass to module 125a for respectively point cloud (370a) and image (370b) classification and segmentation. Once classified, the two data are merged together by module S380. Thereafter, the merged data is passed to module 125n for the final features extraction process which will produce a set of; vector files
• the client 122 may gather the required data packets and dispatch the data packets to various sub-modules that can take care of the direct georeferencing, via a dataGeorefHandler module 124 on the server 121 , and to the metrics productions through the metricXHandler module(s) 125a-125n atS370, S380, S390 on the client 122.
• StreamReader module 123 may include, but not be limited to, a stream Reader3D module 1 1 1 , a streamReader2D module 1 12, streamReaderPOS module 123c, and sysConfig Reader 123d.
• the StreamReader module 123 can interface with the imaging module (the 3D sensor 1 1 1 or the 2D sensor 1 1 1 ), for 3D and 2D sensors respectively, and POS S330.
• the 2D image data is transmitted from the 2D sensor 1 12 to the streamReader2D module 1 12
• the position and/or orientation related data is transmitted from the POS 130 to the streamReaderPOS module 123c
• the StreamReader module 123 can receive or collect the 3D image data, 2D image data, the position and/or orientation related data, and the system installation information respectively from the 3D sensor 1 1 1 , the 2D sensor 1 12 and the POS 130 (S350).
• a dataGeorefHandler module 124 may perform an actual georeferencing of the
• 3D image data and 2D image data S360.
• coordinates in a sensor frame are transformed to coordinates in a mapping frame, for example ECEF coordinates.
• a LiDAR equation and/or photogrammetric equations can be used to perform the transformation.
• the dataGeorefHandler module 124 can perform an automatic post-processing of a trajectory using Applanix Cloud service or PosPAC script (not shown).
• another sub-module can be added to perform a strip alignment at the end of the 3D image data and the 2D image data.
• the strip alignment may also improve any inaccurate boresight angle estimation.
• the dataGeorefHandler module 124 can perform the strip alignment.
• the Metricsl Handler module 125a can be a classifier for the 3D image data and the 2D image data.
• indices such as NDVI, SR and PRI are extracted or generated from bands ratio, in most of the time the 2D image data is processed using remote sensing image analytics to segment the 2D image data.
• intensity value of the point cloud may be corrected geometrically and calibrated radiometrically. Thereafter, an octree is computed so that the plurality of cells have the same size, regardless of the number of points included in each cell. Thereafter, a normal is computed for each of the data points, and eigenvalues and/or eigenvectors are created for each of the plurality of cells based on the computed normal.
• the normal may be a vector perpendicular of the best-fitting plane for the set of the data points.
• the eigenvalues and/or eigenvectors for each of the plurality of cells are for covariance matrix created from each of the data points’ nearest neighbours.
• the geometrical attribute may include, but not be limited to, linearity, planarity, scattering, omnivariance, anisotropy, recitropy, local curvature, normal values, intensity, elevation and delta elevation with regards to an Above Ground Elevation (AGE).
• AGE Above Ground Elevation
• the point cloud is segmented according to the geometrical attribute.
• the geometrical attribute it may be appreciated that at least one of geometrical features, elevation, normal and intensity, homogeneity/variation or value can be used to segment the point cloud.
• the data points in each cell are classified according a predetermined set of classes, for example the ASPRS LiDAR classes. Thereafter, the data points may be meshed and used as level zero for the AGE estimation. Thus, each of the data points can have a class and the AEG elevation value.
• the 2D image data and the 3D image data are then be merged together (S380).
• the metrics2Handler module 125b can re-compute the at least one geometrical attribute for each of the plurality of cells and/or re-classify each of the data points, based on the merged image data.
• the metrics2Handler module 125b can select and perform either re-computation of the geometrical attribute or re- classification of each of the data points.
• the metrics2Handler module 125b can add other geometrical attributes, for example vegetation health, chlorophyll concentration, etc. Addition of the geometrical attributes to the data points are limited to first return point as the 2D image data come from 2D sensor 1 12 which is a passive sensor and does not have penetrating capability in term of vision.
• the 2D image data of vegetation index is used to re-enforce and affine the vegetation delineation in the point cloud.
• the fusion between 2D image data and 3D image data may rely on photogrammetric and ray tracing principles.
• this step of S380 can be omitted.
• the metricsNHandler module 125n is used to apply each of the data points having a class and the AEG elevation value to a generation of the geographical information (S390). In this manner, the metricsNHandler module 125n can generate the geographical information that the user desires.
• the system 100 can generate the geographical information without the 2D image data. If there is no 2D image data, the step of 1 12 and elements of 320a, 360c and 370b can be omitted from Fig. 4.
• the processor 120 can also operate as a post processor.
• the 2D image data and/or 3D image data are received from one or more image files, for example 2D image file and/or 3D image file, instead of 2D sensor 1 12 and/or 3D sensor 1 1 1.
• the 3D image file for example a raw LiDAR file, may be an RXP file and accessible through a predetermined library.
• the POS file may be either a post-process trajectory in Smooth Best Estimate Trajectory (SBET) format or‘file rnav_Mission1 .out’ format generated from Applanix PosPAC software.
• SBET Smooth Best Estimate Trajectory
• the latter file format can be used in case a real-time positioning service such as RTK (Real-time base station corrections) or RTX (Real-time satellite corrections) is used.
• the latter file may be the same format as the SBET file format, but does not require a post- processing applied to it.
• the 2D image file it may depend on the type of 2D sensor 1 12.
• the raw 2D images may be first post-processed image using the SBET trajectory, and then this image may be fed to the processor 120. If a CMOS sensor is used as the 2D sensor 1 12, the 2D images with their time tags can be directly fed into the processor 120.
• Fig. 5 shows a flow diagram regarding a generation of geographical information in accordance with some embodiments of the present disclosure.
• the processor 120 receives a selection of a type of the geographical information to be generated from the user. Although not shown, the processor 120 can suggest various types of geographical information, and the user can select at least one geographical information that the user desires to generate.
• the processor 120 extracts at least one feature associated with the selected geographical information.
• the feature may include, but not be limited to, at least one of geographical feature or object feature.
• the geographical information may include, but not limited to, High Definition maps resulting from 3D feature extraction, classification and segmentation, road and pavement surface condition monitoring and maintenance, change detection in natural and build environments, or plant information such as tree health.
• the geographical information may include, but not be limited to, under-canopy mapping information or emergency response information.
• the processor 120 can generate an HD map as the geographical information as follows.
• the generating process may correspond to S260 in figure 2.
• the processor 120 extracts at least one feature associated with the selected geographical information.
• the feature to be extracted may be related to road condition and may include, but not limited to, road centreline, road marks, kerbs, traffic and light posts, signs, cables, etc.
• the extracting process may correspond to S250 in figure 2.
• the processor 120 may be used with a dual scanner mobile mapping system.
• the dual scanner mobile mapping system may be composed of two LiDAR sensors, a controller unit, a camera trigger box, a Global Navigation Satellite System (GNSS) receiver, and a 360° spherical camera.
• GNSS Global Navigation Satellite System
• the processor 120 may be used with an Applanix AP60, a VMX system (a Riegl’s MMS VMX-1 HA that has 2 Riegl VUX-1 HA), and a Ladybug5 camera provided from FLIR.
• the processor 120 can use a 360° rotating LiDAR sensor as the 3D profiling sensor (3D sensor 1 1 1 ).
• the processor 120 can further use the 2D profiling sensor (2D sensor 1 12).
• the profiling sensors can collect 2D slices. The 2D slices can be stacked to generate the 3D image data.
• the profiling sensors can be used to extract near-vertical 2D slices at the back of the vehicle which will allow the centreline extraction.
• the point cloud may be analysed using a Piecewise Linear Time Series Segmentation approach using a sliding windows by calculation of a moving average, to find corners or breaks in the point cloud.
• This analysis process may correspond to S240 in figure 2. Applying semantic analysis on their relative positions with respect to the vehicle and/or between each other’s, the road edges and/or kerbs are identifiable and the centreline is deduced.
• the point cloud may be generated using an image data. This generation process may correspond to S230 in figure 2.
• an analysis of an accumulation of the position and/or orientation related data generated by the POS 130 may help to detect the road centreline.
• the detection of the centreline may provide a set of directional vectors that can be used as a guide to slide a sliding analytic window in the right direction, to reduce the processing time.
• the accumulation can also be converted into polylines that can be thereafter saved as Shapefile vector files. The saved files can be re-inspected later.
• the kerbs are the same pattern, for example an alternate black and grey (natural colour of the stone or concrete) strips. Therefore, the extraction of the kerbs is based on geometrical property as well as physical attribute, for example its colour and reflectance. The extraction may be done after the centreline detection, as each side of the road has been extracted. Once the edge of the road has been found, the reflectance may be analysed to find the limits of the kerbs. Road marks may be extracted based on intensity attribute. The road point cloud may be extracted from the overall point cloud using the kerbs limit information. From this sub-point cloud, i.e. the extracted point cloud, an intensity threshold may be applied and the remaining points may be clustered using a Euclidean distance filter. Thereafter, each cluster is analysed for its shape and label.
• an extracted cylindrical based feature may be either manmade object such as light post, traffic light, etc., or a tree.
• a slice of data is extracted between 1 to 1.3 metre AGE. This slice of data is then divided into smaller parts to reduce processing time. For example, this slice of data is clustered using a Euclidean distance segmentation algorithm. For each cluster, a cylinder is fitted in the clustered slice of data, and a fitting score is outputted. The fitting score is used to assess the type of shape.
• the fitting score may be an actual residual normal, which may be an actual RMSE (Root Mean Square Error) of the orthogonal distance between the best-fitting planes estimated from the points and the cylinder.
• RMSE Root Mean Square Error
• the cluster shows a cylindrical form which means either a tree or some kind of manmade pole. If the fitting score is low, then the cluster is not the cylindrical shape and can be put aside for further processing.
• the processor 120 can then extract other slices of data point every metre or 2 metres upward (until no more points are available) and perform the cylinder fitting. If the subsequent fittings show high scores and similar cylinder properties have all centred at around the same X, Y geographical centre as well as a consistent number of points within each cluster, then the processor 120 assumes that this feature is a manmade feature. Further processing is then performed by extracting a buffer area around the centroid of the lowest cluster.
• the slice sequence’s attributes may then be used to establish the type of object (based on height, diameter, etc.) using a machine learning approach.
• a Quantitative Structure Model (QSM) algorithm is applied to the buffered point cloud.
• This algorithm may produce a watertight mesh from the trunk and branches network. From this mesh, the processor 120 may evaluate the bounding box and adjust the size of the buffer zone. For this new buffer zone, the processor 120 can evaluate the height of the tree with respect to AGE and its crown diameter by fitting a circle or ellipsoid.
• the fitting object may be decided.
• the fitting object may include, but not be limited to, cylinder, circle and/or ellipsoid.
• a library which performs the fitting may offer multiple types of object for the fitting and output the fitting score.
• the processor 120 may then evaluate the volume of the tree (trunk and branches network) using the mesh produced by QSM algorithm. From the volume and tree species information, for example from third source, the processor 120 may evaluate weight and/or carbon content of the tree.
• Every object is then saved in a Geographical Information System database where it can be displayed and analysed further.
• results in real-time or near real-time are stored with appropriate timestamps and in the I MU/BODY reference frame. This may allow to refine final position of the features as if the user wishes a more accurate position, he may perform a post-process the trajectory to obtain a SBET file and apply it to the points of interest.
• ⁇ Tree Health (as shown in Fig. 5)
• the processor 120 can generate a tree health information as the geographical information as follows. This generation process may correspond to S260 in figure 2.
• the processor 120 can perform a mapping of trees along roadside, parks and housing block areas and assess their health.
• the geographical information may be used by an arborist to determine at least one of the following information: tree species based on images, tree girth measurement at height of predetermined metre from the ground, estimated tree height and crown size, tree location of all trees mapped, or a point feature indicating the centroid of the trunk at height of predetermined metre from the ground.
• two types of kinematic systems integration may be used.
• the first system may be from a vehicle such as a car, and the other system may be from a backpack.
• the processor 120 may be used with the Riegl’s MMS VMX-1 HA.
• the Riegl’s MMS VMX-1 HA may be composed of two Riegl VUX-1 HA, a controller unit and a camera trigger box, an Applanix
• the second system may be composed of the processor 120 (with smaller footprint), a Riegl miniVUX-1 UAV with an APX20 from Applanix, the Theta V, a 360° spherical camera from Ricoh, and a micro- CAS I 1920 from Itres.
• the processing workflow may be identical with the HD map as described above. This means getting the 3D and 2D data from the 3D and 2D imagers, then georeferenced then using trajectory information (S501 and S505). A point cloud can undergo some geometric (S503) and radiometric (S504) corrections. It is to be appreciated that the point cloud may be generated in the same manner as in figures 3 and 4. S503 and S504 may correspond to S240 in figure 2. However, for this application an additional 2D sensor 1 12, for example a hyperspectral imager, may assist in identifying a vegetation in the scene through the analysis of hyperspectral vegetation indexes such as NDVI, SRI, etc (S51 1 ).
• the LiDAR data as 3D image data may be classified using the methodology presented in the HD map as described above. Thereafter, the 2D image data from the hyperspectral imager may be radiometrically corrected using Itres’ RCX library (S507), and then be geometrically projected using the trajectory information (S508)Finally, the calibrated and georectified image data (S510), for example calibrated and georectified 2D image data, may undergo an atmospheric correction (S509).
• the distances between the sensor and the objects are minimal, thus no atmospheric correction may be applied, since the trees are on the ground.
• a value of the NDVI is computed (S51 1 ). Based on the computed value of the NDVI, the features may be extracted and/or determined. This extraction and/or determination process may correspond to S250 in figure 2. Low values of NDVI, for example 0.1 and below, may indicate to non-vegetated areas. Moderate values of NDVI, for example 0.2 to 0.3, may indicate shrub and grass. High values of NDVI, for example 0.6 to 0.8, may indicate trees and forests. The reference values can be determined by the user and/or the processor 120.
• 3D and 2D images, S510, S51 1 , S506 and S504 may be fused together based on their timestamp information.
• the fusion process may comprise point cloud colorization (S521 ), classification (S522) based on geometrical, colour and intensity attributes, and finally segmentation (S523).
• tress may be extracted from the point cloud, generating a tree mesh (S524) for volume estimation using a QSM algorithm.
• a Carbon content can be estimate using tree species information provided by local arborist and the tree volume (S526).
• a tree health indices can be estimate using Vegetation Index at S51 1.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Remote Sensing (AREA)
• Radar, Positioning & Navigation (AREA)
• Theoretical Computer Science (AREA)
• Electromagnetism (AREA)
• Data Mining & Analysis (AREA)
• Computer Networks & Wireless Communication (AREA)
• Multimedia (AREA)
• Software Systems (AREA)
• Geometry (AREA)
• Artificial Intelligence (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Life Sciences & Earth Sciences (AREA)
• Bioinformatics & Cheminformatics (AREA)
• Evolutionary Computation (AREA)
• Bioinformatics & Computational Biology (AREA)
• Evolutionary Biology (AREA)
• General Engineering & Computer Science (AREA)
• Computer Graphics (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Automation & Control Theory (AREA)
• Spectroscopy & Molecular Physics (AREA)
• Image Processing (AREA)
• Image Analysis (AREA)
• Processing Or Creating Images (AREA)

Priority Applications (7)

Applications Claiming Priority (2)

Publications (1)

Family

ID=70055977

Family Applications (1)

Country Status (8)

Cited By (1)

Families Citing this family (4)

Citations (5)

Family Cites Families (7)

• 2019
  • 2019-09-18 KR KR1020207008297A patent/KR20210067979A/ko not_active Ceased
  • 2019-09-18 GB GB2019895.8A patent/GB2589024A/en not_active Withdrawn
  • 2019-09-18 JP JP2020517585A patent/JP2022511147A/ja active Pending
  • 2019-09-18 WO PCT/SG2019/050466 patent/WO2020072001A1/en not_active Ceased
  • 2019-09-18 AU AU2019352559A patent/AU2019352559A1/en not_active Abandoned
  • 2019-09-18 CN CN201980004687.7A patent/CN113168714A/zh active Pending
  • 2019-09-18 SG SG11202009873RA patent/SG11202009873RA/en unknown
  • 2019-10-02 TW TW108135719A patent/TW202022808A/zh unknown

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events