KR102520676B1 - Tree species detection apparatus based on camera, thermal camera, GPS, and LiDAR and Detection method of the same - Google Patents

Abstract

The present invention relates to a technology for detecting various information on trees using a vehicle sensor system which integrates four sensors: a thermal imaging camera, a camera, a LiDAR, and GPS. By installing the camera, the thermal imaging camera, the GPS, and the LiDAR sensor on the top of the vehicle and using fused high-dimensional data, tree species detection, which requires skilled manpower, is replaced with deep learning. The advantage of maximizing reliability and accuracy is realized by directly measuring structural information of trees, which was previously measured indirectly, through the LiDAR sensor.

Description

Tree species detection apparatus based on LiDAR and camera collection information and tree information detection method using the same {Tree species detection apparatus based on camera, thermal camera, GPS, and LiDAR and Detection method of the same}

The present invention relates to a technology for detecting various information of a tree using a vehicle sensor system integrating four sensors: a thermal imaging camera, camera, LIDAR, and GPS.

Vegetation investigation is an essential precedent for identifying characteristics such as systematization of plant communities distributed in green and forest ecosystems, species composition, and plant sociological structure. Vegetation surveys give priority to direct observation, and records must be faithfully performed through field records or photography and recording.

The paper on which information obtained during vegetation survey is written is generally called field field (or survey table). The paper yard used at this time has several problems. In order to evaluate vegetation, it is necessary to go through the process of directly inputting field records manually recorded in the field into a computer. In this process, it takes additional time to transfer and input information after returning, and input errors that may occur during the input process, concerns about loss and damage of the field field due to delay in input, time to compare paper field information and computer input information after input, etc. Occurs.

In particular, the method to secure data on urban-scale vegetation or tree species existing in a city is mainly to observe the location and type of target tree species within the city by injecting survey personnel. Identification of tree species existing in a city has a problem that consumes a lot of time and money.

Recently, a technology to analyze the greening space of a city has been introduced through aerial photography, and a technology to determine the distribution of vegetation has been disclosed (Korean Registered Patent No. 10-1150510), but even in this case, accurate identification of tree species Overclassification cannot be implemented. In particular, it is limited to grasping the vegetation structure of the three-dimensional structure of the greening area, and the operating cost of the aircraft operated to secure basic data is very expensive, and it is difficult to re-shoot because it is difficult to modify the flight route, and it is difficult to take pictures whenever necessary. There is a disadvantage that the practicality is very low as a problem. Furthermore, in the case of aerial photography, since only the upper part of the tree species is photographed, there is a limit to accurately identifying the tree species. Accordingly, there is a growing need for a technology capable of quickly realizing more accurate identification and distribution of tree species.

Accordingly, in Korea Patent Registration No. 10-2308456, the present applicant proposes a technique for detecting tree species by mounting a lidar sensor on a vehicle, and a method for automatically determining tree species by applying a deep learning model. .

However, in order to manage tree information on a city scale, tree information is not limited to the classification of tree species, and detailed structural information (height, chest height, crown height, basement height, crown width, etc.) and tree state information ( tree temperature, location information, etc.) are required together.

In the fast and autonomous method of collecting city-scale tree species data, the existing method is a method in which a person observes the target tree species one by one. The downside is that it consumes manpower. In addition, skilled manpower is required to classify tree species, and tree structure information such as height and breast height has been measured using an indirect measurement method, so the measured values are not accurate.

The technology of Korean Patent Registration No. 10-2308456 of the present applicant provides an excellent effect of detecting tree species by saving labor and time, but calculates detailed tree structure information (tree structure information such as tree temperature, height, chest height, etc.) Therefore, the need for technology development that can derive detailed tree information is emerging.

Korean Patent Registration No. 10-2308456 Korean Patent Registration No. 10-1150510 Korean Patent Registration No. 10-1561669

The present invention has been made to solve the above-mentioned needs, and an object of the present invention is to install a camera, a thermal imaging camera, a GPS, and a lidar sensor on top of a vehicle to detect tree species requiring skilled manpower by using converged high-dimensional data. is replaced by deep learning, and reliability and accuracy can be maximized by directly measuring tree structure information, which used to be indirect surveying, through lidar sensors, while scanning urban vegetation through vehicles and automatically By detecting information, temperature information, and location information, it is to provide a technology that can reduce the labor and time required for landscaping management compared to the existing method of direct observation by a person, and expand the scale of management to the city scale.

As a means for solving the above problems, in an embodiment of the present invention, as shown in FIGS. 1 to 4, the step of generating location information of data, a mobile traveling device (S: vehicle) that moves on the ground; a sensor module (100) mounted on the mobile traveling device (S) to form image data, point cloud data, thermal image, and location data for the surrounding scenery; Tree species information calculation module 200 that fuses image data, point cloud data, and thermal images provided from the sensor module 100 to form convergence data including information applied to tree species classification, and calculates information about tree species. ); and an urban street tree database 300 for storing and classifying structural information on tree species calculated by the tree species information calculation module 200, wherein the tree species information calculation module 200 includes image data, point cloud data, and columns. By fusing image images, using the fusion data including information applied to tree species classification as an input variable to the deep learning model, tree species information including temperature information and location information of trees is generated, and tree point cloud data is obtained from the point cloud data. To provide a tree information detection device applying lidar and composite camera modules to obtain tree structure information including any one of tree height, breast height diameter, crown width, basement, and crown height by classifying do.

In addition, in another embodiment of the present invention, the above-described tree information detection device is applied to detect tree information, but the camera module 110 and lidar module 120 mounted on the mobile traveling device (S) moving on the ground Step 1 of receiving data provided from the sensor module 100 including the thermal image module 130 and the GPS module 130 and pre-processing it in the data acquisition unit 210; a second step of fusing image data, point cloud data, and thermal images through calibration in the calibration unit 220 to form convergence data to be used for deep learning; a third step of generating, in the deep learning unit 230, tree species information including temperature information and location information of trees by using the fusion data including information applied to tree species classification as an input variable to a deep learning model; Step 4 of classifying tree point cloud data from the point cloud data in the tree structure information calculation unit 240 and calculating tree structure information including tree height, breast height diameter, crown width, height below ground level, and crown height of the tree; The tree species information derived from the deep learning unit 230, the tree structure information derived from the tree structure information calculation unit 240, and the tree surface temperature and location information obtained from the data acquisition unit 210 are utilized to make a street tree. 5 steps to perform database construction; Including, it is possible to provide a tree information detection method applying LiDAR and a composite camera module.

According to an embodiment of the present invention, by installing a camera, a thermal imaging camera, a GPS, and a lidar sensor on top of a vehicle and using converged high-dimensional data, tree species detection requiring skilled personnel is replaced with deep learning, and indirect surveying is performed. It has the effect of maximizing reliability and accuracy by directly measuring the structure information of the fallen tree through the lidar sensor.

In addition, according to the present invention, by scanning urban vegetation through a vehicle and automatically detecting tree species, structure information, temperature information, and location information of street trees, labor and time required for landscaping management are reduced compared to the existing method directly observed by a person. and the scale of management can be expanded to the city scale.

Furthermore, various sensing data are secured through lidar, camera modules, and thermal imaging camera modules mounted on mobile equipment (vehicles), and through analysis of these data, accurate tree species are detected and data that derives structural information of the tree itself. and a method for calculating tree structure information. Through this, it is possible to accurately detect tree species and derive tree structure information at a very low cost compared to existing aircraft-based systems, and since it is a vehicle-based system, it is possible to repeatedly photograph the same area, and to photograph the front of the tree species to be detected. Through this, the advantage of being able to more accurately detect tree species and derive tree structure information is realized.

In addition, in order to improve the error rate of the deep learning application technique for each lidar and camera data of the previous registered patent (Korean Patent Registration No. 10-2308456) of the previous applicant, two deep learning models are integrated into one to manage the entire processor And it facilitates improvement, and improves the performance of the deep learning model for detecting species information by additionally including the temperature information derived from the thermal image camera data as a deep learning variable.

In addition, in the present invention, by using high-dimensional data (point cloud including RGB and temperature information) that fuses data between sensors through calibration in the input stage, the problem of different dimensions of data generated by each sensor [point cloud] Clouds are 3D, images are 2D; When the information contained in each dimension (point cloud: intensity, image: temperature, RGB) is included, the difficulty of constructing a single deep learning model derived from 4-dimensional, 3-dimensional] has been resolved to improve efficiency.

1 and 2 are block diagrams of a system configuration of an apparatus for detecting tree information (hereinafter, referred to as 'the present invention') to which a lidar and composite camera module are applied according to an embodiment of the present invention.
3 and 4 are flowcharts illustrating a process of performing tree species detection using the system of FIG. 1 .
5 to 14 are conceptual diagrams for explaining the configuration of FIGS. 1 and 2 .

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content will be thorough and complete and the spirit of the present invention will be sufficiently conveyed to those skilled in the art.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and unless explicitly defined in this application, they should not be interpreted in ideal or excessively formal meanings. don't

1 and 2 are block diagrams of a system configuration of an apparatus for detecting tree information (hereinafter, referred to as 'the present invention') to which a lidar and composite camera module are applied according to an embodiment of the present invention. 3 and 4 are flowcharts illustrating a process of performing tree species detection using the system of FIG. 1 .

1 to 4, the present invention is a mobile traveling device (S: vehicle) that moves on the ground;

A sensor module 100 mounted on the mobile traveling device S to form image data, point cloud data, thermal image, and location data of the surrounding scenery, image data provided from the sensor module 100, Convergence data including information applied to tree species classification is formed by converging point cloud data and thermal images, and is calculated by the tree species information calculation module 200 and the tree species information calculation module 200 that calculates information on tree species. It is configured to include a city street tree database 300 that stores and classifies structural information on tree species to be classified.

In particular, the tree species information calculation module 200 converges image data, point cloud data, and thermal images, and sets fusion data including information applied to tree species classification as an input variable to a deep learning model to obtain tree temperature information and Tree species information including location information is generated, tree point cloud data is classified from the point cloud data, and tree structure information including any one of tree height, diameter at chest height, crown width, basement height, and crown height is acquired. and the acquired information can be built into the city street tree database 300. This process can be regularly and conveniently managed on a city scale and monitored by updating the tree database.

As shown in FIG. 5, the mobile traveling device S of the present invention includes at least one LiDAR (Light Detection and Ranging) sensor module 120 and a camera module 110 disposed in a vehicle, and a thermal image Although the method of mounting the sensor module 100 containing the camera module 130 and the GPS module 140 is described, this is according to an embodiment, and various fields of detecting objects by converging lidar data and images It can be applied to, and it will be obvious that it can be applied to various mobile devices (motor, drone, helicopter, etc.), not limited to automobiles. However, in a preferred embodiment of the present invention, as a method for deriving tree structure information at low cost, implementation by applying a car will be described as an application example.

The camera module 110 is mounted on the upper part of the mobile traveling device (S: vehicle), generates an image of the surrounding environment (front) of the vehicle, and acquires the data of the tree information calculation module 200 from the generated image. An operation of transmitting to the unit 210 is performed. Here, the video includes a plurality of images, and each image refers to 3D image data including color information similar to that of the human eye. The camera module may be implemented in various types of image capturing devices as long as it can generate an image of an image around the vehicle.

The thermal imaging camera module 130 is a device module capable of capturing and transmitting a thermal image of a tree included in the surrounding landscape outside the vehicle, and includes a thermal imaging camera and a transmission device capable of transmitting EZ images captured through the thermal imaging camera. It is a concept that

The lidar module 120 is mounted on the top of the vehicle and emits a laser toward the periphery (front) of the vehicle. In an embodiment of the present invention, a lidar module (eg, Velodyne lidar) having a structure capable of rotating 360 degrees in addition to the front can be applied, but is not limited thereto. In the case of the lidar module applied to the present invention, it can be implemented by combining known lidar sensors in various ways, and it is possible to obtain a 3D point cloud by combining a module structure capable of rotating 360 degrees or at least one lidar module. It is also possible to implement it as a structure. In the lidar module 120, the laser emitted by the lidar sensor may be scattered or reflected and returned to the vehicle. The lidar sensor represents distance information measured using a laser in the form of a set of points in 3D space, and lidar data including such distance information (hereinafter, referred to as 'point cloud data'. ) is transmitted to the tree species detection module 200. For example, the LIDAR sensor may obtain information about physical characteristics of a target located around the vehicle based on a change in time, intensity, frequency, and polarization state of the laser beam.

The GPS module 140 calculates the location information of the vehicle, and derives and combines location information about the data sensed by the camera module 110, lidar module 120, and thermal image camera module 130 described above. It enables the formation of positional data that can be used.

In the present invention, by installing the above-described camera, thermal imaging camera, GPS, and LiDAR sensor on top of the vehicle and using converged high-dimensional data, detection of tree species requiring skilled personnel is replaced with deep learning, and It is possible to provide a device and system technology that can improve the existing traditional method by directly measuring structural information through a LIDAR sensor.

To this end, the sensor module 100 of the present invention, as shown in FIGS. 1 and 2, is mounted on the mobile traveling device (S), image data for the surrounding scenery, point cloud data, thermal image, It performs a function of forming location data.

The sensor module 100 includes a camera module 110 that collects image data of the surrounding scenery in which the vehicle is driving, a lidar module 120 that collects point cloud data of the surrounding scenery in which the vehicle is driving, and a vehicle driving A thermal imaging camera module 130 that collects thermal image data of the surrounding landscape, and a GPS that collects location information of a driving vehicle and provides information for deriving the location information of the data on the surrounding landscape. It may be configured to include module 140 .

Image data, point cloud data, and thermal image data provided by the sensor module 100 described above are transmitted to the tree species information calculation module 200, and in the tree species information calculation module 200, the sensor module 100 The provided image data, point cloud data, and thermal image are fused to form fusion data including information applied to tree species classification, and information on tree species is calculated.

To this end, the tree species information calculation module 200, as shown in FIGS. 1 and 2, mounted on the mobile traveling device (S), the camera module 110, lidar module 120, thermal image The data acquisition unit 210 for receiving and pre-processing the data provided from the sensor module 100 including the module 130 and the GPS module 130, and the image of the surrounding scenery provided from the sensor module 100 Calibration unit 220 that fuses data, point cloud data, and thermal images through calibration to form convergence data to be used for deep learning, and inputs the convergence data including information applied to species classification to the deep learning model The deep learning unit 230 generates tree species information including temperature information and location information of trees by using a tree height, height at breast height, crown width, and height based on the 'tree point cloud' separated from the point cloud data. , a tree structure information calculating unit 240 that calculates the height of a tree can be configured.

The data acquisition unit 210 performs a function of pre-processing and classifying data provided from various sensors, and specifically, a device that removes image data generated by a camera lens and distortion of the edge of a thermal image. A first processing unit 212, a second processing unit 214 that removes the noise of the point cloud data by applying a statistical outliner method, and projecting the 3D point cloud data into a 2D xy plane lattice, It may be configured to include a third processing unit 216 that removes the ground by utilizing the maximum height, minimum height, and average height.

Describing the function of the data acquisition unit 210, first, as a function of the first processing unit 212 that pre-processes data, the exemplary diagram of FIG. 6 ((a) before distortion correction (b) after distortion correction) As in, it performs a function of performing distortion correction on the image data of the surrounding landscape and the thermal image data. This can be performed by applying an algorithm that removes the distortion (X-->X') of the edge of the image caused by the camera lens using the camera's internal parameter values.

Next, shown in FIG. 7 is a function of the second processing unit 214 that removes the point cloud outliner, and shows an image (a) before noise removal (b) after noise removal. Since a lot of noise occurs due to the characteristics of point cloud data, it can be implemented through an algorithm that removes it by a statistical outlier removal method.

Also, shown in FIG. 8 is a function of the third processing unit 216 of the present invention, which shows an image (a) before point cloud ground removal and (b) after point cloud ground removal. Since the ground point cloud must be removed before the next process, after projecting the 3D point cloud into a 2D xy plane lattice, applying an algorithm that removes the ground using the maximum height, minimum height, and average height of the points of each lattice make it possible to implement

The image data, point cloud data, and thermal image data preprocessed in the above-described data acquisition unit 210 are fused through calibration in the calibration unit 220 to form fusion data to be used for deep learning.

That is, the calibration unit 220 calibrates data acquired from different sensors to generate merged data. In general, in the case of an image, it is 3-dimensional data by adding 2-dimensional spatial information and 3 channels of R, G, and B. In the case of a point cloud, 4-dimensional data is obtained by adding 1 channel of intensity (laser return intensity) to 3-dimensional spatial information. will be composed For these data, high-dimensional data with 5 channels of intensity, R, G, B, and temperature is created in 3-dimensional spatial information through inter-sensor calibration. Furthermore, it is possible to generate location information of all data acquired through calibration of the GPS module 130 and the LIDAR module 120.

9 is an exemplary diagram for explaining the function of the calibration unit 220. The calibration unit 220 of the present invention performs inter-sensor calibration by projecting a point cloud onto a camera image and a thermal image. .

10 is a concrete application of the function of the calibration unit 220 described above in FIG. 9, (A) for image data, thermal image data, and LIDAR data obtained from each sensor obtained through the data acquisition unit, (B ) It shows the result of forming high-dimensional fusion data through data fusion between sensors. In this case, (a) shows the result of fusing lidar data and thermal image data, and (b) shows the result of fusing lidar data and camera data.

In addition, in the present invention, as shown in FIG. 11, the above-described calibration unit 220 (A) fuses lidar data with respect to data detected by various sensors (eg, camera image data) to obtain species information It is possible to obtain point cloud data of individual trees with

The deep learning unit 230 performs a function of generating tree species information including temperature information and location information of trees by using the fusion data including information applied to tree species classification as an input variable to a deep learning model.

The tree structure information calculation unit 240 calculates the height, chest diameter, crown width, height of the tree, and crown height based on the 'tree point cloud' separated from the entire point cloud data. That is, it is possible to derive tree structure information by using the individual tree point cloud derived as a result of tree species information through deep learning.

To this end, the tree structure information calculation unit 240, as shown in FIGS. 1 and 2, the effort derivation unit 242 for deriving the effort through the difference between the maximum height and the minimum height of the tree point cloud, the tree A crown height derivation unit 244 that separates the crown portion and the column portion by calculating the point density based on the z value of the point cloud, and derives the crown height and the basement through the difference between the maximum height and the minimum height. A crown width derivation unit 246 for deriving the crown width by calculating the longest side using the information, a chest height derivation unit for deriving the diameter of the breast height by deriving the cross section of the derived column part and calculating the diameter through a circle detection algorithm ( 248).

The process of deriving the tree structure information calculated through the tree structure information calculation unit 240 can be implemented by programming as shown in FIG. 12, and the above information is classified and stored as a database as tree structure information as shown in FIG. be able to That is, the data in FIG. 13 can be constructed with the city street tree database 300 described above in FIG. 1 . That is, tree species information derived from the deep learning unit 230, tree structure information derived from the tree structure information calculation unit 240, and tree surface temperature and location information obtained from the data acquisition unit 210 of the present invention. It is possible to build a street tree database by using .

A method of detecting tree information using lidar and composite camera modules by applying the above-described present invention will be described with reference to FIGS. 3 and 4.

The above-described method for detecting tree information using lidar and composite camera modules to which the present invention is applied includes a camera module 110 mounted on a mobile traveling device (S) moving on the ground, a lidar module 120, A first step of receiving data provided from the sensor module 100 including the thermal image module 130 and the GPS module 130 and pre-processing it in the data acquisition unit 210 is performed. In this case, the observation data are lidar point cloud (3D spatial information), camera image, thermal image, and GPS location information.

With respect to the observation information, the data acquisition unit removes image data generated by the camera lens and distortion of the edge of the thermal image (first processing unit 212), and applies a statistical outliner method to noise of the point cloud data. and removes it (second processing unit 214), and further, after projecting the 3D point cloud data into a 2D xy plane lattice, using the maximum height, minimum height, and average height of the points of each lattice to remove the ground (first 3 processing unit 216) may be performed.

Thereafter, as a two-step process, a step of forming convergence data to be used for deep learning by fusing image data, point cloud data, and thermal images through calibration in the calibration unit 220 may be performed. Among observation data, LiDAR point cloud, camera image, and thermal imaging camera are converged through calibration to build high-dimensional convergence data to be used for deep learning. Tree species classification of roadside trees is a task of high difficulty to be performed with only one sensor's acquisition data, so data from three sensors are converged to build high-dimensional data that includes more information to be used for tree species classification, thereby maximizing its reliability. do. In addition, by converging GPS and point clouds, location information is included in the point cloud.

Next, as a three-step process, a process of obtaining tree species information through deep learning can be performed. That is, the deep learning unit 230 generates tree species information including temperature information and location information of trees by using the fusion data including information applied to tree species classification as an input variable to the deep learning model.

After constructing a deep learning model for detecting tree species by utilizing the high-dimensional convergence data constructed in the above-described step 2, tree species information is generated and transmitted to the tree information database in the present invention described above. At the same time, only the tree point cloud is separated from the entire point cloud. As described above, this tree point cloud is used for tree structure information analysis. In addition, the tree point cloud includes temperature and location information through sensor-to-sensor calibration, so that the tree temperature and location information can be provided to the city street tree database.

Next, in a four-step process, the tree structure information calculation unit 240 classifies the tree point cloud data from the point cloud data, and includes the height of the tree, diameter at breast height, crown width, basement height, and crown height of the tree. A step of calculating tree structure information is performed. That is, by using the tree point cloud acquired in the above-mentioned 3 steps, an algorithm for calculating the height, chest diameter, crown width, height of the tree, and crown height is constructed. These are classified as tree structure information and then transmitted to the city street tree database.

Thereafter, the tree species information derived from the deep learning unit 230, the tree structure information derived from the tree structure information calculation unit 240, and the tree surface temperature and location information acquired from the data acquisition unit 210 are utilized. 5 steps are performed to build a street tree database.

In the present invention, data convergence technology of a vehicle-based sensor system is implemented to build an effective city-scale tree database, and a method for constructing an integrated tree database is provided to extract various characteristics of trees and maintain a database of urban green spaces. can be used In addition, by applying the sensor system to a person-attached type or a drone-attached type rather than a vehicle-attached type, an effective tree database can be established not only in city streets, but also in parks and forests.

In particular, by scanning urban vegetation through a vehicle and automatically detecting tree species, structure information, temperature information, and location information of street trees through the proposed algorithm, labor and time required for landscaping management are reduced compared to the existing method directly observed by humans. and the scale of management can be expanded to the city scale.

Furthermore, the present invention can be used for various applications such as planning and modeling of urban space, facility management, and mapping in a city 3D model. In more advanced forms, it can also be used in other technology fields including robotics, remote sensing, environmental assessment, entertainment industry, indoor modeling and more.

In addition, functional configurations and execution operations applied to a system executing the above-described tree information detection method according to the present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented with any number of hardware or/and software components that perform specific functions. For example, the present invention relates to integrated circuit components, such as memory, processing, logic, look-up tables, etc., that can perform various functions by means of the control of one or more microprocessors or other control devices. can be hired

Similar to components of the present invention that may be implemented as software programming or software elements, the present invention may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, including Python ), C, C++, Java, assembler, and the like, programming or scripting languages may be implemented. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, the present invention may employ conventional techniques for electronic environment setting, signal processing, and/or data processing. Terms such as “~ module”, “~ unit”, “mechanism”, “element”, “means”, and “component” may be used broadly and are not limited to mechanical and physical components. The term may include a meaning of a series of software routines in association with a processor or the like.

As described above, the technical spirit of the present invention has been specifically described in the preferred embodiment, but the above preferred embodiment is for explanation and not for limitation. As such, those skilled in the art will understand that various embodiments are possible through the combination of the embodiments of the present invention within the scope of the technical spirit of the present invention.

100: sensor module
200: tree information calculation module
210: data acquisition unit
220: calibration unit
230: deep learning unit
240: tree structure information calculation unit
300: city street tree database

Claims (11)

A mobile traveling device that moves on the ground (S: vehicle);
A camera module 110, a lidar module 120, and a thermal imaging camera module 130 that are mounted on the mobile traveling device S and form image data, point cloud data, thermal image, and location data for the surrounding scenery. ) and the sensor module 100 including the GPS module 140; Convergence of image data, point cloud data, and thermal images provided by the sensor module 100 forms fusion data including information applied to tree species classification, and calculates information on tree species based on the fusion data Tree species information calculation module 200; A city street tree database 300 for storing and classifying structural information on tree species calculated by the tree species information calculation module 200;
The species information calculation module 200,
High-dimensional convergence with five channels of intensity, R, G, B, and temperature in three-dimensional spatial information through calibration between sensors by projecting point cloud data, which is information provided by the sensor module 100, onto camera image data and thermal images A calibration unit 220 that generates data and generates location information of all data acquired through calibration of the GPS module 140 and the lidar module 120;
a deep learning unit 230 for generating tree species information including temperature information and location information of trees by using the convergence data as an input variable to a deep learning model;
A tree structure information calculation unit 240 that calculates height, chest diameter, crown width, basement height, and crown height based on the 'tree point cloud' separated from the point cloud data constituting the convergence data; including,
Tree information detection device applying LIDAR and composite camera module.
The method of claim 1,
The species information calculation module 200,
A camera module 110 mounted on the mobile traveling device (S); lidar module 120; thermal imaging module 130; GPS module 140; data acquisition unit 210 for receiving and pre-processing the data provided from the sensor module 100 including; Further comprising,
Tree information detection device applying LIDAR and composite camera module.
Tree information detection device applying LIDAR and composite camera module.
The method of claim 2,
The data acquisition unit 210,
a first processing unit 212 that removes image data generated by the camera lens and distortion of edges of the thermal image;
a second processing unit 214 for removing noise of point cloud data by applying a statistical outliner method;
a third processing unit 216 that projects the 3D point cloud data onto a 2D xy plane lattice and then removes the ground using the maximum height, minimum height, and average height of the points of each lattice;
including,
Tree information detection device applying LIDAR and composite camera module.
delete delete delete The method of claim 3,
The city street tree database 300,
The tree species information derived from the deep learning unit 230, the tree structure information derived from the tree structure information calculation unit 240, and the tree surface temperature and location information obtained from the data acquisition unit 210 are utilized to make a street tree. to build a database,
Tree information detection device applying LIDAR and composite camera module.
Tree information is detected by applying the tree information detection device of claim 7,
Data provided by the sensor module 100 including the camera module 110, the lidar module 120, the thermal image module 130, and the GPS module 140 mounted on the mobile traveling device (S) moving on the ground. Step 1 of receiving and pre-processing in the data acquisition unit 210;
a second step of fusing image data, point cloud data, and thermal images through calibration in the calibration unit 220 to form convergence data to be used for deep learning;
a third step of generating, in the deep learning unit 230, tree species information including temperature information and location information of trees by using the fusion data including information applied to tree species classification as an input variable to a deep learning model;
Step 4 of classifying tree point cloud data from the point cloud data in the tree structure information calculation unit 240 and calculating tree structure information including tree height, breast height diameter, crown width, height below ground level, and crown height of the tree;
The tree species information derived from the deep learning unit 230, the tree structure information derived from the tree structure information calculation unit 240, and the tree surface temperature and location information obtained from the data acquisition unit 210 are utilized to make a street tree. Including; 5 steps to perform database construction;
The second step is
High-dimensional convergence with five channels of intensity, R, G, B, and temperature in three-dimensional spatial information through calibration between sensors by projecting point cloud data, which is information provided by the sensor module 100, onto camera image data and thermal images A step of generating data and generating location information of all data acquired through calibration of the GPS module 140 and the lidar module 120,
Tree information detection method applying LIDAR and composite camera module.
delete delete A computer-readable recording medium recording a program for executing a tree information detection method applying the lidar and composite camera module according to claim 8 to a computer.

Images