KR102170745B1 - Method of estimating a location by fusing indoor spatial information data - Google Patents

Abstract

The present invention relates to a method for estimating a location through fusion of indoor spatial information data, and using a mobile mapping system (MMS), three-dimensional image information in the building for autonomous driving of a vehicle entering the building, and A first step of obtaining location information; A second step of receiving two-dimensional image information and location information, which are architectural drawing information of the building, from an external server; By matching the three-dimensional image information and location information obtained in the first step with the two-dimensional image information and location information received in the second step, the spatial information in the building for parking the autonomous vehicle is obtained. Including a third step of generating; Including, it is possible to obtain the location information of the vehicle even when a vehicle for autonomous driving enters a building where GPS signals cannot be received, and it is difficult to obtain image information and location information inside the building. It is possible to provide a technology that can supplement spatial information on the boundary area.
According to the present invention, by fusing the building construction drawing including the location information with the image information and location information obtained through the mobile mapping system (MMS) of the autonomous vehicle, the autonomous vehicle entering the building is automatically Parking can be enabled, and by fusing the processed two-dimensional building construction drawings with three-dimensional raw data, it is possible to supplement the image information and location information incorrectly obtained by the attached installations in the building. It includes the effect of generating indoor spatial information, and by fusing architectural drawings with 3D point cloud data, it automates the process of constructing 3D spatial information, which is mostly manual, so that the update of spatial information in the building is faster. Includes the possible effects.

Description

Location estimation method through fusion of indoor spatial information data {METHOD OF ESTIMATING A LOCATION BY FUSING INDOOR SPATIAL INFORMATION DATA}

The present invention relates to a method of estimating the position of a vehicle by fusing indoor spatial information data.

Recently, automobile-related technologies have begun to focus on technologies that prioritize the safety of pedestrians and drivers in the development direction for improving vehicle stability or for simple driver's convenience. To this end, the Advanced Driver Assistance System (ADAS), a technology that controls the vehicle's mechanical devices by recognizing some of the numerous situations that can occur while driving by itself and judging the situation has been proposed. In addition to helping the driver in the complex vehicle control process, it is ultimately possible to enable the vehicle to run autonomously.

The driving of the vehicle starts in the parked state and ends again in the parked state. The automatic parking system designed for parking convenience and safety during the driving process is one of the early technologies of ADAS, and it can be strictly referred to as a parking steering assistance system.

The above-described automatic parking is useful for the elderly and the inexperienced driving, and for drivers with relatively low spatial sense ability, a blind spot that is not visible during parking further increases the difficulty of parking. In fact, it is said that the stress felt when parking is greater than the stress of driving on the road that a novice driver feels.

However, the automatic parking system has the advantage of reducing the difficulty of parking by using the accuracy of a mechanical device, but there are many difficulties in a method of estimating a location for automatic parking of an autonomous vehicle.

In general, a technology that enables vehicles for autonomous driving to locate the current position of the vehicle based on the equipped global positioning system (GPS) and to automatically drive and park based on the current position is proposed. Has been done. However, when a vehicle enters a parking lot provided in a building or under the building to drive and park, the GPS receiving module provided in the vehicle cannot receive signals from GPS satellites.

Accordingly, there is a need to develop a technology capable of calculating the current position of the autonomous vehicle even if a GPS signal is not received from the autonomous vehicle, and automatically driving and parking operations from the calculated current position to an empty parking space.

『Republic of Korea Patent Publication No. 10-1560984, Title of Invention: Automatic Parking System, (Notification: Oct. 27, 2015, Patent Holder: Sungkyunkwan University Industry-Academic Cooperation Foundation)』, the location information is entered based on the landmark A technical configuration of an automatic parking system for controlling driving and parking of an autonomous vehicle by generating driving information to an available parking area of a vehicle is disclosed.

However, the above-described prior art is a configuration in which an autonomous vehicle is guided to a parking area along a preset driving path based on a landmark, and an automatic parking system including three-dimensional spatial information is separately installed in the building. It seems difficult to change the route due to unexpected variables such as obstacles or pedestrians on the guided route as well as the hassle.

In addition, there is also a problem in that it is difficult to generate more accurate spatial information in the building because image information or location information acquired by installations such as guide signs may not be accurate.

Patent Document (0001) 『Korean Registered Patent Publication No. 10-1560984, Title of Invention: Automatic Parking System, (Notice Date: October 27, 2015, Patent Holder: Sungkyunkwan University Industry-Academic Cooperation Foundation)』

The present invention has been made to solve the above-described problem, and an object of the present invention is to provide a technology capable of obtaining location information of a vehicle even when a vehicle for autonomous driving enters a building where GPS signals cannot be received.

In addition, there is another purpose to provide a technology that can supplement spatial information on a boundary area where it is difficult to obtain image information and location information within a building.

The problem to be solved by the present invention is not limited to the above-described problem, and other technical problems that are not mentioned will be clearly understood by those of ordinary skill in the technical field to which the present invention belongs from the following description.

In order to achieve this object, in a method for estimating the location of an autonomous vehicle for automatic parking according to an aspect of the present invention, autonomous driving of a vehicle entering a building using a mobile mapping system (MMS) A first step of obtaining three-dimensional image information and location information in the building for; A second step of receiving two-dimensional image information and location information, which are architectural drawing information of the building, from an external server; By matching the three-dimensional image information and location information obtained in the first step with the two-dimensional image information and location information received in the second step, the spatial information in the building for parking the autonomous vehicle is obtained. It includes; a third step of generating.

And, acquiring 3D image information from LiDAR (Light Detection And Ranging) and CCD camera (Charge-Couple Device Camera, CCD Camera); And obtaining three-dimensional location information from a global positioning system (GPS) and an inertial navigation system (INS).

In addition, receiving two-dimensional image information and location information in the building from a building information transmission terminal installed in the building; Or receiving two-dimensional image information and location information in a building through short-distance communication or long-distance communication from a GIS (Geographic Information System) server.

In addition, the 3D image information and location information and the 2D image information and location information are combined using an area-based method (Aerial Based Matching), a feature-based method (Feature Based Matching), or a relational method (Relational Matching). It may include; generating spatial information in the building by matching.

In addition, the step of generating a parking path by estimating the position of the autonomous vehicle based on the spatial information in the building generated in the third step; may further include.

In order to achieve this object, in another aspect of the present invention, in a system for estimating the location of an autonomous vehicle for automatic parking, a 3D information collection terminal that acquires 3D image information and location information in a building using a mobile mapping system ; A 2D information collection terminal for receiving 2D image information and image information, which are architectural drawings of a building from an external server; And the image information and location information collected from the 3D information collection terminal, and the image information and location information collected from the 2D information collection terminal to generate a 2D coordinate value, and the 3D image information and location information and the 2 And a data fusion server for generating spatial information of 3D coordinate values by calculating a height value according to a height difference between dimensional image information and location information.

In addition, the 3D information collection terminal may include a configuration of a LiDAR and a CCD camera for acquiring 3D image information, and a configuration of a satellite navigation device and an inertial navigation device acquiring 3D location information.

In addition, the 2D information collection terminal receives two-dimensional image information and location information in the building from a building information transmission terminal installed in the building, or through short-distance communication or long-distance communication from a GIS (Geographic Information System) server. It is possible to receive the image information and location information of.

In addition, the data fusion server generates spatial information in the building by matching the 3D image information and location information with the 2D image information and location information using an area-based method, a shape-based method, or a relationship-based method. can do.

In addition, a location estimation server for generating an automatic parking route by estimating the location of the autonomous vehicle based on spatial information in the building generated by the data fusion server may further include.

The means for solving the above-described problems are merely exemplary and should not be construed as limiting the present invention. In addition to the above-described exemplary embodiments, there may be additional embodiments described in the drawings and detailed description of the invention.

As described above, according to the present invention, the following effects are obtained.

First, by fusing the building construction drawing including location information with the image information and location information acquired through the Mobile Mapping System (MMS) of the autonomous vehicle, automatic parking of the autonomous vehicle entering the building is possible. I can do it.

Second, by fusing the processed two-dimensional building construction drawings with three-dimensional raw data, it is possible to supplement the image information and location information incorrectly acquired by the attached installation in the building, thereby generating more accurate indoor spatial information. Includes possible effects.

Third, by fusing architectural drawings with 3D point cloud data, the process of constructing 3D spatial information, which is mostly manually performed, is automated, so that the spatial information in the building can be updated more quickly.

The effects of the present invention are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a block diagram showing the configuration of a location estimation system through fusion of indoor spatial information data according to an embodiment of the present invention.
2 is a block diagram showing a detailed configuration of a data fusion server according to an embodiment of the present invention.
3 is a block diagram showing a detailed configuration of a location estimation server according to an embodiment of the present invention.
4 is a flowchart illustrating a method of estimating a location through fusion of indoor spatial information data according to an embodiment of the present invention.

Preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings, but technical parts that are already well-known will be omitted or compressed for conciseness of description.

The present invention is roughly divided into a technology for fusing indoor spatial information data for autonomous parking of a vehicle and a technology for estimating the location of an autonomous vehicle through the fused indoor spatial information data.

First, the overall configuration of a system for estimating the location of an autonomous vehicle by fusion of indoor spatial information data will be described.

1 is a block diagram showing the configuration of a location estimation system through fusion of indoor spatial information data according to an embodiment of the present invention.

As shown in Fig. 1, a location estimation system 100 (hereinafter referred to as a'location estimation system') through fusion of indoor spatial information data according to an embodiment of the present invention is an MMS (Mobile Mapping System) installed in an autonomous vehicle. System) to acquire three-dimensional raw data including image and location information inside the building, and fusion of two-dimensional image information and location information of architectural drawings to the obtained three-dimensional raw data, and unprocessed three-dimensional data By easily distinguishing the object or topographic features of the building, it is possible to more quickly and accurately construct 3D spatial information within the building.

Through this, it is possible to easily generate 3D spatial information inside a building where GPS signals are not received, thereby facilitating automatic parking of an autonomous vehicle entering the building.

The location estimation system 100 of the present invention comprises a 3D information collection terminal 110, a 2D information collection terminal 120, a data fusion server 130, and a location estimation server 140 in order to perform the above-described functions. Include.

The 3D information collection terminal 110 is installed in an autonomous vehicle to move along a driving path, and acquires image information of a terrain feature in the driving path and location information of the image. LiDAR 111 for obtaining image information ( Light Detection And Ranging) or an image information collection module composed of a CCD camera 112 (Charge-Couple device Camera, CCD Camera) device, and a satellite navigation device 113 (Global Positioning System, GPS) for obtaining location information, and The image information acquired through the location information collection module composed of the inertial navigation system 114 (Inertial Navigation System, INS) and the location information are transmitted to the data fusion server 130 through the collection information transmission module again.

Here, the configuration of the image information collection module including the LiDAR 111 and the CCD camera 112, and the location information collection module including the satellite navigation device 113 and the inertial navigation device 114 is well known in the prior art. These are well-known essential components used in the system, so a detailed description thereof will be omitted.

In addition, the technical idea of the present invention is not limited to the above-described means for acquiring the image and location information, and other known components are used in the process of acquiring the image and location information. Will be included.

The 2D information collection terminal 120 receives architectural drawing information of a building to acquire 2D image information and location information of a parking space, and is connected to a terminal for transmitting drawings in the building or external servers through wired/wireless communication. At this time, the building may be provided with a building information transmission terminal (not shown) that transmits architectural drawing information, and the 2D information collection terminal 120 is NFC (Near Field Communication) communication, RFID (Radio Frequency Identification) communication, Zigbee ( Zigbee) communication and short-range communication such as Wi-Fi (Wireless-Fidelity) are used to receive architectural drawing information, which is 2D image information, and connect to a GIS (Geographic Information System) server through long-distance communication such as LAN or WAN communication. To receive architectural drawing information.

Here, the architectural drawings collected by the 2D information collection terminal 120 supplement the location information of the satellite navigation device 113 that cannot be collected when the autonomous vehicle enters the building, thereby providing 3D spatial information of the parking space in the building. Makes it easy to build.

The data fusion server 130 combines the image and location information of the 3D and 2D spatial information transmitted from the 3D information collection terminal 110 and the 2D information collection terminal 120 described above, so that the autonomous vehicle can park. It generates 3D spatial information in the driving building.

LiDAR 111 data consists of points in which all data have x, y, and z values, and all point data provides the position of a feature on the ground as a position coordinate in a three-dimensional ground coordinate system. Hereinafter, the means for obtaining image information of the mobile mapping system will be collectively referred to as LiDAR 111.

Using LiDAR(111) data, it is possible to reproduce the appearance of the ground, such as creating a digital surface model (DSM) that normalizes the target area by dividing it into a certain grid or extracting a flat patch. However, the DSM and the flat patch are easy to implement a rough ground appearance, but the data density is relatively low and the clarity is low, and it is crucial to acquire accurate point data at the boundary of buildings, etc. Because it is difficult, it requires fusion with other data.

Accordingly, the two-dimensional image information and the location information of the architectural drawing are matched or fused to compensate for the shortcomings of the LiDAR 111 data to generate spatial information in the building.

The fusion process of 3D spatial information and 2D spatial information performed by the above-described data fusion server 130 will be described again in FIG. 2 to be described later.

The location estimation server 140 determines the location of the vehicle and the parking location, the parking path or the location of obstacles in the space based on the 3D spatial information in the building generated through the data fusion server 130, and determines the size and shape of the vehicle. By modeling, the autonomous vehicle calculates a moving path of the vehicle that can reach the correct parking position by avoiding obstacles, as well as controlling the steering so that the vehicle can move along the generated parking path.

The above-described process of generating a parking route by the location estimation server 140 will be described in detail again in FIG. 3 to be described later.

The user terminal 150 is installed in an autonomous vehicle to provide spatial information for automatic parking and the estimated location and parking route of the vehicle to the user, or a separate terminal, for example, a user's smart phone. It can be provided to users through the App of

Here, the user terminal 150 allows the user to graphically check the 3D and 2D image information and location information received by the data fusion server 130, and the two image information and the location information are created by matching. It is possible to provide a tool that allows a user to modify and supplement spatial information in a building that is being created.

In addition, it is possible to provide a separate tool capable of not only showing the parking route of the vehicle provided by the location estimation server 140 but also correcting the route.

In the following, a technology for fusion of indoor spatial information data through a data fusion server will be described.

2 is a block diagram showing a detailed configuration of a data fusion server according to an embodiment of the present invention.

As shown in Figure 2, the data fusion server 130 of the present invention is a 2D information collection terminal 120 to the unprocessed three-dimensional spatial information obtained through the 3D information collection terminal 110 installed in the autonomous vehicle as described above. By fusing the two-dimensional spatial information, which is the architectural drawing information collected through the system, more accurate three-dimensional spatial information in the building is generated, as well as the location information in the building that cannot receive the location information by the satellite navigation system. Can be provided through.

LiDAR 111 data acquired through the 3D information collection terminal 110 is easy to implement a schematic building shape, as described above, but the data density is relatively low and the clarity is low, and the data from the sensor is definitively acquired. Due to the nature of the method, it is difficult to obtain accurate point data at the boundary line of a building.

Since the architectural drawing obtained through the 2D information collection terminal 120 is a vectorized main boundary line of the building, it can be very easily used to generate spatial information in the building through fusion with the LiDAR 111 data.

In particular, since the architectural drawing clearly expresses the boundary inside the building, as mentioned above, when generating spatial information indoors where it is difficult to receive GPS signals, it is possible to provide location information as well as LiDAR (111) data. It is possible to supplement the spatial information on the boundary area that is difficult to obtain.

However, it is preferable that the corresponding boundary surfaces in the architectural drawing should be reconstructed into the same single layer, and the scale of the architectural drawing and the LiDAR 111 data should be similarly reconstructed.

Since the reconstructed architectural drawing layer has actual horizontal coordinates, it is possible to easily obtain all boundary points of the moving space of people or vehicles in the building.

In this way, by fusing 3D image information and location information of LiDAR (111) data with 2D image information and location information of architectural drawings to create a 3D plane, 3D spatial information in the building consisting of a ceiling surface, a wall surface, and a floor surface It can be built more accurately and easily.

To this end, the data fusion server 130 includes a data preprocessing unit 131, a data matching unit 132, a matching degree evaluation unit 133, a modeling preprocessing unit 134, a height modeling unit 135, and a filtering unit 136. ), the 2D modeling unit 137, and the 3D modeling unit 138.

The data preprocessing unit 131 is to process the 3D image information transmitted from the 3D information collection terminal 110 and the corresponding position information, and the 2D image information transmitted from the 2D information collection terminal 120 and the corresponding position information. Each of them is pretreated.

The preprocessing process of 3D image information, which is LiDAR(111) data, is a process of removing outliers, and the preprocessing process of 2D image information, which is an architectural drawing, is a spatial layer of the part that forms the moving space of a person or vehicle in the entire drawing It is a process of extraction.

First, the preprocessing process of the LiDAR 111 data will be described, and the point density used as a reference for detecting an outlier can be classified into an absolute point density and a relative point density. Absolute point density can determine outliers when the density of the local area is less than the standard compared to the density of the entire data range, and the relative point density is similarly compared to the density of the local area with the standard, but the standard at this time is It means the density of neighboring areas, not the density of data. Relative point density is not an outlier, but data acquisition is small depending on the shape of the structure, so it can prevent errors that are judged as outliers, but it is not practical because the density is calculated by individually checking the number of surrounding points for all points. . Therefore, it is desirable to use an outlier detection algorithm in which the absolute point density and the relative point density are considered in combination.

On the other hand, when explaining the preprocessing process of architectural drawings, architectural drawings are spaces that accommodate installations such as beams (columns) that support loads, walls that divide space, wiring, as well as spatial information that people or vehicles can move. Since it contains information on spaces where people or vehicles cannot move, it is not suitable for direct application to the modeling process. Through this, the extracted spatial information for the movement of a person or vehicle is in the form of a two-dimensional polygon, and the coordinates of the boundary points are extracted using GIS commercial software to obtain the location coordinates of the architectural drawing.

The data matching unit 132 matches the pre-processed 3D image information and 2D image information, so that the object recognition and modeling information contained in the LiDAR 111 data can be elaborated by clarifying the boundary information of the object using architectural drawings. do. These two data matching must precede the unification of the coordinate system. However, even if the coordinate system is unified, a minute error may occur, and the minute error may be large in the building modeling process, so that the two data can be completely overlapped through a plurality of transformation processes. To this end, coordinate transformation is performed by selecting the conjugated point of the two data.

Specifically, matching can be defined as making or finding something that is equal or similar, and establishing a correspondence. Matching techniques applied in various fields can be largely classified into area-based matching, shape-based matching, and symbolic matching, and each method can be compared with a similarity measurement and a primitive used. have. Objects used in shape-based registration are interest points and edges. A typical matching sequence is to extract objects, select possible matching pairs of entities, and estimate transform factors or final matching pairs. In the case of object estimation, the generated 3D segment is grouped in 2D and 3D. The final 3D position may use the original edge pixels rather than a line segment.

As a constraint used for matching, there is an epipolar geometry. Line segments extracted from the two data generally appear in different shapes as a whole, and since the common part of the two segments exists partially, line segment matching is somewhat more difficult than point matching. However, the line segment shown in one image must have a corresponding line segment at least partially within the quadrilateral of the other data defined by the conjugated geometry and the height condition of the building. Knowing the height range of the object, the search space must exist in the quadrilateral by the minimum and maximum heights. Any line segment that is partially present in this quadrilateral can be a corresponding line segment. The edge segment to be matched must have a conjugate geometry and a somewhat large (close to vertical) pier. Otherwise, the image expression information error may increase a bias in the estimated disparity.

Edge registration using a 3D image can be classified into two types. In the first approach, an edge of one image and an edge set of another image are directly compared, and the best candidate is selected according to the similarity function. The similarity compares edge properties such as direction, length, and area information related to the edge. In the second approach, the corresponding edge is found by structural matching. In structural matching, we find the transformation equation of two structural descriptions. Structural technology consists of geometrical and topological information between objects and objects.

Incidentally, the image matching method is an area-based method based on the similarity of brightness values to a local area (Aerial Based Matching), a feature-based method based on the similarity of shapes between the extracted preceding objects (Feature Based Matching), and the extracted There are methods such as relational matching that considers the relationship between objects. However, the matching of the 3D image information and the 2D image information is because the shape of the image acquired by the difference of the observation point, slope or undulation is considerably different even if the image information is acquired for the same part. It's hard to bet. In particular, the shape-based method or the relationship-based method is relatively robust against the problem of distorting and projecting images according to slopes or undulations, but it takes a lot of time to extract objects and has a problem of low reliability of the extracted objects. Although the algorithm is simple and operates relatively quickly, the method is highly affected by distortion.

In the embodiment of the present invention, the process of matching 3D image information and 2D image information is assumed and described, centering on the shape-based method, but is not limited thereto, and other methods including the area-based method or the relationship-based method. It will be apparent that the use of the image matching method is also included in the technical idea of the present invention.

The consistency evaluation unit 133 may increase modeling accuracy for a moving path of an autonomous vehicle by evaluating the result of matching the LiDAR 111 data and the architectural drawing. The result of performing data matching is not always successful due to various reasons. Therefore, it is necessary to verify the quality of the matching result to confirm whether the conjugated point corresponding to each matching target point generated through matching is correctly selected.

A typical method of evaluating the matching quality is to calculate the correlation coefficient of the brightness value, for example, indicating the similarity of the brightness value of the local area around the target point and the corresponding conjugated point using a region-based method.

For example, the standard deviation of the estimated value of the variance factor and the estimated value of the conjugate position calculated from the residual, which is the difference between the observed value and the calculated value, is calculated. The estimated value of the variance factor represents the average difference between the matching target point of the reference image and the brightness value around the conjugated point finally determined corresponding thereto. The smaller the estimated value of the variance factor is, the greater the probability that the conjugated point, which is determined to have a smaller difference in brightness, approaches the true value.

In this way, when 3D image information and location information, 2D image information, and location information corresponding thereto are matched, a modeling process for generating spatial information in the building is performed using the matched two image information.

The modeling preprocessing unit 134 performs a preprocessing process for performing a 3D modeling process using the matched two data, and the point cloud data of a path representing the autonomous driving space of a vehicle in which 3D image information and 2D image information are matched is It is difficult to visually check indoor facilities and noise due to external structures such as ceilings, walls, and floors surrounding the space of the path, and inefficient manual work is required to remove them.

Therefore, it is necessary to measure the height of the indoor space and remove noise in order to generate a drawing of the driving route. Accordingly, the parking space is divided at regular intervals, and 3D coordinate values of point data in the divided space are stored.

Point cloud data obtained through LiDAR 111 surveying and then matched with 2D image information are mainly generated in a Cartesian coordinate system. Since these orthogonal coordinate values are stored in the form of 3D coordinate values, and points generated at defined point intervals store 3D coordinates of nearby point data, the parking space is a box-shaped unit space having the same width. It has the effect of dividing into.

The height modeling unit 135 generates height values of two two-dimensional matched data that do not have a height value, and the floor and ceiling are mainly planar bodies in the parking space, and points having minimum and maximum height values respectively at the same location. It consists of data. The height of the interior space is defined by the difference between the height of the ceiling and the floor model. In order to measure the indoor height, point data having the maximum height value and the minimum height value are extracted from the point of the plane, and a floor plan model is constructed using the RANSAC (RANdom and SAmpling Consensus) algorithm.

Briefly about the RANSAC algorithm, RANSAC is an algorithm that finds an inlier to build a model defined from point cloud data including noise. In order to construct a plan model of the ceiling, the maximum height value is calculated from each plane point. Branch point data is extracted, and point data having a minimum height value is extracted to create a floor plan model. RANSAC creates a flat model by repeating the hypothesis and test steps. In the hypothesis stage, a sample of a certain size is constructed from the extracted point data, and three points are randomly selected to generate a planar model. The plane model is determined by functions and variables, and in the test step, the adequacy of the model is evaluated by calculating the error of point data based on the plane model.

When point data exist within a certain boundary value from the created model, it is classified as a normal value and is expressed as a Consensus Set (CS).

Since parking spaces contain noise of various sizes and shapes, it is possible to have multiple CSs when creating a plan model of a ceiling or floor. In height modeling, it is assumed that the CS with the maximum size represents the ceiling or the floor. The generated ceiling and floor plan model calculates the height of the floor and the ceiling of the corresponding point using the plan coordinates as input variables, and the height of the corresponding point is the difference value between the two values.

The filtering unit 136 removes noise in the generated parking space, and the wall surface in the parking space is mainly a vertical plane and is composed of point data between the minimum height value and the maximum height value. On the other hand, obstacles such as speed bumps and parking blocks of the driving path are classified as noise, and most of these noises exist on the floor and do not include point data having a maximum height value. Accordingly, an offset space having a certain distance from the ceiling plane model is created and set as a reference for noise identification.

In the noise removal process, point data between the floor and ceiling plan models are extracted from each plane point, and the presence or absence of point data in the offset space is checked. If point data exists in the offset space, it is regarded as a wall, and if point data does not exist, it is regarded as noise, and noise is removed by removing point data between the floor and ceiling plan models.

The 2D modeling unit 137 generates a plan view using an outline of the point group data projected on the 2D plane using an extraction grid. However, since the outline of the plan, extracted using the grid, is composed of a number of line segments and has a serrated shape, a refinement process is required for standardization of the drawing.

The refining process consists of a classification process and a normalization process, and the classification is a process of classifying the segments of the initial plan into segments belonging to the same plane. Normalization is a process of adjusting the segments of the plan view reconstructed in the classification process to satisfy the constraints such as parallelism, orthogonality, and intersection.

The 3D modeling unit 138 is generated using the floor plan built through the 2D modeling unit 137 and the height value and noise data measured by the modeling preprocessor 134. In the pretreatment process, plan models of the ceiling and floor are created to measure the height values. The input variables of the plan models are the plane coordinate values of the floor plan corners, and the heights of the ceiling and the floor of the corresponding point are calculated. The 3D frame model may be generated using the height calculated at the corner of the 2D plan view.

However, the 3D wireframe model represents only the outer surface of the parking space. The noise data identified in the preprocessing process not only describe the parking space, but also represent the shape and position of objects in the space, so it is combined with the wire frame model and used for detailed modeling.

In the following, a technology for estimating the location of an autonomous vehicle through a location estimation server will be described.

3 is a block diagram showing a detailed configuration of a location estimation server according to an embodiment of the present invention.

The location estimation server 140 more accurately constructs spatial information in buildings that cannot receive GPS signals by fusing 3D image information obtained through LiDAR 111 data with 2D image information of the structured architectural drawings. By clearly estimating the location information of my self-driving vehicle, it enables autonomous parking of the driving vehicle.

Parking is a lot of attention to general drivers as well as beginner drivers. This is because the relationship between the position of the vehicle and the distance to nearby landmarks, the steering angle and the expected course of the vehicle must be determined based on the driver's senses. Parking in a narrow space may cause a collision with other vehicles or obstacles due to a driver's instantaneous position determination error or steering wheel manipulation error. In order to relieve the economic loss caused by this and the pressure on the driver's parking, it is necessary to determine the possible path to the parking position by determining the exact vehicle location and the distance to nearby landmarks when parking, and to help the precise control of the steering wheel. do.

As described above, in order for the driving vehicle to park automatically, first, identify the position of the vehicle, the parking position, and the position of the obstacle, and second, calculate the movement path of the vehicle to reach the correct parking position by avoiding the obstacle, and third. In this case, the steering wheel must be controlled so that the vehicle can move along the generated path.

In order to perform the above-described function, the location estimation server 140 of the present invention includes a flat map generation unit 141, an obstacle detection unit 142, a vehicle modeling unit 143, and a path generation unit 144. do.

The flat map generation unit 141 maps a parking space including a moving route of a vehicle in a building using the matched image information generated by the 2D modeling unit 137 or the 3D modeling unit 138, The position of the parking line is expressed on a plane coordinate system based on the positions of the LiDAR 111 and the CCD camera 112 installed in the vehicle. Through this, it is possible to easily express the relationship between the exact parking position of the vehicle and the current vehicle position.

This expression technique has a great advantage in sequential image input. As the vehicle moves, the primary image input through the camera continuously changes. However, when a three-dimensional world is received as a two-dimensional image, a lot of calculations are required to calculate the position of the current vehicle or to compare the previous image information with the current image information. On the other hand, the planar map generator 141 of the present invention converts the image information into a plane coordinate system and then updates the previous image information by overlaying the transformed image on the moved position. It can be easily updated.

The obstacle detection unit 142 detects a moving path of an autonomous vehicle and an obstacle in a parking space to determine a movable area and an available parking area.

In order to detect an obstacle using an image, the type of the obstacle must first be defined. The main obstacle when parking will be a vehicle parked in another parking space. In addition to the vehicle, there may be people and other objects, but the vehicle in motion or the vehicle being parked is regarded as a major obstacle and the obstacle is detected.

A shadow is generated at the bottom of the vehicle, and an edge can be extracted using a point having a property different from that of the road surface, and a vehicle on the road can be detected by the edge. In this way, the presence or absence of a vehicle in the parking line is determined to recognize the entire interior of a parking line as an obstacle.

First, in order to detect the parking line, a binary image representing the candidate of the parking line is created from the original image through a color threshold. In order to effectively remove sporadic noise, a clean image without noise is obtained by successively performing Erosion and Dilation, which are Morphology techniques.

In order to find the linear component in the image, a chain code is used to find the linear component, and the parking section, which is a rectangular area, is made into a binary image using the linear component.

In order to detect a vehicle that is an obstacle, an edge is extracted for an area within each parking segment. The number of edges in the area is calculated by applying a threshold value to the extracted edges, and a parking section in which the number of edges is greater than a certain number is determined as a parking section in which an obstacle exists.

The vehicle modeling unit 143 calculates a motion model capable of calculating the location of the vehicle, and in order to accurately identify the location of the vehicle on a flat map, a model capable of calculating the location of the vehicle through the steering angle and speed of the vehicle is required. . Since the center of the vehicle's turning radius is the center of the two rear wheels, it can be calculated in the same way as calculating the rotation trajectory of a four-wheel vehicle using a model of a two-wheeled vehicle that considers only the steering angle of the front wheel and the length of the vehicle.

The path generator 144 generates a parking path of the autonomous vehicle on a flat map in which an obstacle is recognized, and it is preferable that the path is generated based on a change in steering angle according to an obstacle, a distance, and a distance. Specifically, it is desirable to have a distance from obstacles on each path, a change in a steering angle when traveling along a path, and a distance to a final position to generate a close path.

In this embodiment, a method of generating an automatic parking path of a driving vehicle has been described using the point that the vehicle turns based on a certain rotational center, but this is only one embodiment, and other well-known path generation methods are It goes without saying that what is applied and used selectively is also included in the technical idea of the present invention.

On the other hand, the core technical idea of the present invention is to create spatial information inside the building by fusing 3D image information and 2D image information, so that the location of the vehicle in the automatic parking route including the exact position and parking position of the vehicle in the spatial information For the purpose of estimating, the detailed description of the steering process of controlling the steering wheel so that the vehicle can move along the generated path among the requirements required for the vehicle to park automatically as described above will be omitted. .

Hereinafter, a method of estimating the position of an autonomous vehicle by fusing indoor spatial information data according to the present invention will be described.

4 is a flowchart illustrating a method of estimating a location through fusion of indoor spatial information data according to an embodiment of the present invention.

As shown in FIG. 4, the method of estimating a location through fusion of indoor spatial information data of the present invention is largely divided into a method of fusion of indoor spatial information data and a method of estimating the position of an autonomous vehicle.

First, a method of fusing indoor spatial information data (S1000) will be described below.

Through the 3D information collection terminal 110 installed in the autonomous vehicle, 3D image information including image information of the moving path and parking space of the vehicle in the building and the location information is collected and transmitted to the data fusion server 130. S1100)

At this time, the 3D information collection terminal 110 includes an image information collection module including a LiDAR 111 or a CCD camera 112, a satellite navigation device 113 (GPS) and an inertial navigation device 114 (INS). It includes the configuration of the location information collection module.

In addition, the 2D information collection terminal 120 installed in the autonomous vehicle collects architectural drawing information of the building through wired/wireless communication with a building information transmission terminal or a GIS external server in the building, and transmits it to the data fusion server 130. (S1200)

At this time, the 2D information collection terminal 120 collects architectural drawing information from the terminal for transmitting information in the building using short-range communication such as NFC communication, RFID communication, Zigbee communication, and Wi-Fi communication, as well as LAN or WAN communication. It collects architectural drawing information from an external server using remote communication such as.

Meanwhile, the data fusion server 130 receiving 3D image information and 2D image information from the 3D information collection terminal 110 and the 2D information collection terminal 120 combines the 2D image information with the 3D image information to move the autonomous vehicle. Generate spatial information of the route and parking space (S1300)

In order to match 3D image information and 2D image information, the data preprocessing unit 131 removes outliers from the received 3D image information, and divides the moving path and parking space of the autonomous vehicle in the architectural drawing. The spatial layer is extracted, and the received two image information is preprocessed (S1310).

Here, in the preprocessing process of 3D image information, it is preferable to detect outliers by considering the absolute point density and relative point density in combination, and the preprocessing process of 2D image information is to divide a space according to whether a person or a vehicle is moving. The spatial layer for moving or parking is extracted.

The preprocessed 3D image information and 2D image information not only unify the coordinate system, but also perform coordinate transformation by selecting a conjugated point, so that the 3D image information and the 2D image information can be completely overlapped (S1320).

For example, when matching two image information using a shape-based matching method, an object is first extracted, a possible matching entity pair is selected, and a transformation factor or a final entity correspondence pair is estimated.

When the two image information are matched, the matching result is evaluated by checking whether the conjugate point is correctly selected in order to increase the degree of matching. (S1330)

Specifically, a correlation coefficient of, for example, a brightness value indicating the similarity of the brightness value of the local region around the target point and the corresponding conjugated point is calculated using the region-based method.

When 3D image information and 2D image information go through a matching process in which the coordinate system is located, a modeling process is performed using the matched two image information (S1400).

In order to perform the modeling process, noise included in the two matched images is removed, and the parking space is divided into box-shaped unit spaces (box thresholds) having the same width by using orthogonal coordinate values (S1410).

Point data having the maximum height value and the minimum height value are extracted from the matched two image information, the height difference is used to calculate the height value of the space, and a plan model of the ceiling and the floor is constructed using the RANSAC algorithm. S1420)

Here, the RANSAC algorithm creates a plane model by repeating the hypothesis and test steps, and in the hypothesis step, a sample of a certain size is constructed from the extracted point data, and three points are selected to generate the plane model. The plane model is determined by functions and variables, and in the test step, the adequacy of the model is evaluated by calculating the error of point data based on the plane model.

Obstacles including speed bumps and parking blocks on the driving path or parking path are identified as noise and removed. (S1430)

Here, the obstacle does not have a maximum height value or a minimum height value, point data having a maximum height value and a minimum height value are regarded as a wall surface, and data not having a maximum or minimum height value are classified as obstacles.

A plan view is generated by extracting the outline of the point cloud data projected on the two-dimensional plane formed by matching the two image information (S1440).

Here, the generated plan view requires a refinement process for formalization, the refinement process consists of a classification process and a normalization process, and the classification is a process of classifying line segments of the initial plan into line segments belonging to the same plane. Normalization is a process of adjusting the segments of the plan view reconstructed in the classification process to satisfy the constraints such as parallelism, orthogonality, and intersection.

A 3D frame model is generated using the floor plan constructed in the 2D modeling process, the height value of the space, and noise data calculated using the difference between the maximum height value and the minimum height value (S1450).

Here, the detailed modeling of the space can be performed by expressing the outer surface of the parking interior space using the above-described plan view and the height value on the space, and expressing the shape and position of the object in the space using noise data.

Next, a method of estimating the position of the autonomous vehicle (S2000) will be described below.

When 3D image information and 2D image information are matched to generate 3D spatial information of the vehicle's driving path and parking space, the autonomous vehicle estimates and calculates the position information of the autonomous vehicle instead of the position information of the satellite navigation device. A route for parking of an autonomous vehicle is created through the location information.

To this end, using the matched image information, a parking space including a moving route of a vehicle in a building is mapped onto a flat surface (S2100).

Here, the position of the parking line is expressed on a plane coordinate system based on the positions of the LiDAR 111 and the CCD camera 112 installed in the vehicle. Through this, the relationship between the current position of the vehicle and the parking position can be clearly and easily expressed. Specifically, the primary image input through the camera continuously changes. However, when a three-dimensional world is received as a two-dimensional image, a lot of calculations are required to calculate the position of the current vehicle or to compare the previous image information with the current image information. This hassle can be easily updated flattened image information by overwriting the converted image to the moved position through correlation comparison with the update of previous image information.

When a flat map for parking is generated, a process of determining a movable area and an available parking area is performed by detecting the moving path of the autonomous vehicle and obstacles in the parking space (S2200).

Here, an edge may be extracted using a shadow generated at the bottom of an obstacle, and a vehicle on the road may be detected by the edge. In this way, the presence or absence of a vehicle in the parking line is determined to recognize the entire interior of a parking line as an obstacle.

Meanwhile, in order to calculate a motion model capable of calculating the position of the vehicle, a model capable of calculating the position of the vehicle through the steering angle and speed of the vehicle is generated (S2300). Specifically, only the steering angle of the front wheel and the length of the vehicle are considered. Using the model of a two-wheeled vehicle, the rotation trajectory of a four-wheeled vehicle is obtained.

When a flat map including obstacle information is generated and a vehicle motion model is calculated, a parking path for an autonomous vehicle is generated (S2400).

Here, the parking path is to create a path based on the change of the steering angle according to the obstacle, distance, and distance. Specifically, the distance from the obstacle must be far, the change of the steering angle must be small, and the distance to the final position must be a close path. Let it be created.

As described above, a detailed description of the present invention has been made by an embodiment with reference to the accompanying drawings, but since the above-described embodiment has been described with reference to a preferred example of the present invention, the present invention is limited to the above embodiment. It should not be understood as being, and the scope of the present invention should be understood as the following claims and the concept of equality.

100: Location estimation system through indoor spatial information data fusion
110: 3D information collection terminal
111: LiDAR
112: CCD camera
113: satellite navigation system
114: inertial navigation system
120: 2D information collection terminal
130: data fusion server
131: data preprocessor
132: data matching unit
133: Matching degree evaluation unit
134: modeling preprocessor
135: height modeling unit
136: filtering unit
137: 2D modeling unit
138: 3D modeling unit
140: location estimation server
141: flat map generator
142: obstacle detection unit
143: vehicle modeling unit
144: path generation unit
150: user terminal

Claims (10)

In the method of estimating the location of an autonomous vehicle for automatic parking,
A first step of acquiring 3D image information and location information in the building for autonomous driving of a vehicle entering the building using a mobile mapping system (MMS);
A second step of receiving two-dimensional image information and location information, which are architectural drawing information of the building, from an external server;
By matching the three-dimensional image information and location information obtained in the first step with the two-dimensional image information and location information received in the second step, the spatial information in the building for parking the autonomous vehicle is obtained. Including; a third step of generating;
The third step,
The three-dimensional image information and location information and the two-dimensional image information and location information are matched using an area-based method (Aerial Based Matching), a feature-based method (Feature Based Matching), or a relational method (Relational Matching). Including; to generate spatial information in the building by,
In the third step, a ceiling plane model and a floor plane model are constructed, an offset space having a certain distance from the ceiling plane model is created and set as a noise identification criterion, but the presence or absence of point data in the offset space is checked, If the point data does not exist in the offset space, it is regarded as noise, and noise is removed by removing point data between the ceiling plane model and the floor plane model.
Location estimation method through indoor spatial information data fusion. The method of claim 1,
The first step,
Acquiring 3D image information from LiDAR (Light Detection And Ranging) and a CCD camera (Charge-Couple Device Camera, CCD Camera); And
Obtaining three-dimensional location information from a global positioning system (GPS) and an inertial navigation system (INS); including
Location estimation method through indoor spatial information data fusion. The method of claim 1,
The second step,
Receiving two-dimensional image information and location information in the building from a building information transmission terminal installed in the building; or
Receiving two-dimensional image information and location information in a building through short-distance communication or long-distance communication from a GIS (Geographic Information System) server; including
Location estimation method through indoor spatial information data fusion. delete The method of claim 1,
Generating a parking route by estimating the location of the autonomous vehicle based on the spatial information in the building generated in the third step; further comprising
Location estimation method through indoor spatial information data fusion. In the system for estimating the position of an autonomous vehicle for automatic parking,
A 3D information collection terminal that acquires 3D image information and location information in a building using a mobile mapping system;
A 2D information collection terminal for receiving 2D image information and location information, which are architectural drawings of the building, from an external server; And
The image information and location information collected from the 3D information collection terminal and the image information and location information collected from the 2D information collection terminal are matched to generate a 2D coordinate value, and the 3D image information and location information and the 2D Includes; a data fusion server for generating spatial information of 3D coordinate values by calculating a height value according to the height difference between image information and location information,
The data fusion server,
By using an area-based method, a shape-based method, or a relationship-based method, the 3D image information and location information and the 2D image information and location information are matched to generate spatial information in the building,
The data fusion server constructs a ceiling plane model and a floor plane model, creates an offset space having a certain distance from the ceiling plane model and sets it as a noise identification criterion, but checks the presence or absence of point data in the offset space, If the point data does not exist in the offset space, it is regarded as noise, and noise is removed by removing point data between the ceiling plane model and the floor plane model.
Location estimation system through indoor spatial information data fusion. The method of claim 6,
The 3D information collection terminal,
Including LiDAR and CCD cameras that acquire three-dimensional image information,
Characterized in that it comprises a satellite navigation device and inertial navigation device for acquiring three-dimensional position information
Location estimation system through indoor spatial information data fusion. The method of claim 6,
The 2D information collection terminal,
Receives two-dimensional image information and location information in the building from the building information transmission terminal installed in the building, or
Characterized in that it receives two-dimensional image information and location information in a building through short-distance communication or long-distance communication from a GIS (Geographic Information System) server.
Location estimation system through indoor spatial information data fusion. delete The method of claim 6,
A location estimation server for generating an automatic parking route by estimating the location of the autonomous vehicle based on spatial information in the building generated by the data fusion server; further comprising
Location estimation system through indoor spatial information data fusion.

Images