KR102686274B1 - Apparatus and method for estimating location based on holistic matching using semantic segmentation image and semantic point cloud map - Google Patents

Abstract

An apparatus and method for location estimation through holistic matching based on a semantic segmented image and a semantic point cloud map are disclosed, and a location estimation method through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application is disclosed. , receiving an image input for a target space, which is the surrounding space for a predetermined moving object, inputting the image based on a first artificial intelligence model previously learned to perform object-based semantic segmentation on the input predetermined image. It may include generating a semantic segmented image corresponding to and estimating location information of the moving object by matching the semantic segmented image with a pre-built semantic point cloud map.

Description

Location estimation device and method through holistic matching based on semantic segmentation image and semantic point cloud map {APPARATUS AND METHOD FOR ESTIMATING LOCATION BASED ON HOLISTIC MATCHING USING SEMANTIC SEGMENTATION IMAGE AND SEMANTIC POINT CLOUD MAP}

This application relates to an apparatus and method for location estimation through holistic matching based on semantic segmented images and semantic point cloud maps.

The term smart car is a convergence of electrical, electronic, and information and communication technologies with traditional machine-centered automobile technology, and is used to improve driver convenience and vehicle safety by various industrial entities such as automobile manufacturers, parts manufacturers, and ICT companies. It has been continuously developed with a focus on these aspects, and these smart car-related technologies specifically include safety and safety systems such as Advanced Driver Assistance System (ADAS), Autonomous Vehicle, and Cooperative Safety System. It can be categorized into related fields and convenience-related fields such as connected cars.

In particular, intelligent cars equipped with ADAS (Advanced Driver Assistance Systems) or autonomous vehicles use various sensors (e.g. cameras, GPS, radar, LiDAR, etc.) to recognize the surrounding environment of the driving vehicle. It should be possible to secure the safety of passengers, pedestrians, and surrounding vehicles and improve driving convenience by monitoring the situation around the vehicle in real time with high accuracy through the acquired sensing data.

In particular, LiDAR is a sensor that measures the distance to a specific target with a laser, determines the shape of the target object, and recognizes the road environment around the car as a three-dimensional shape. LiDAR has a low error in measuring the distance to surrounding objects. By providing precise measurement performance, it has the advantage of being applicable to various vehicle-linked cognitive systems such as obstacle recognition and dynamic object tracking.

However, in the case of a map-based location estimation algorithm using a conventional 3D LiDAR sensor, high accuracy at the centimeter level is guaranteed, but it is applicable only when the vehicle is equipped with an expensive LiDAR sensor, so it is not applicable to mass-produced vehicles. There are limitations that make it difficult to apply.

The technology behind this application is disclosed in Korean Patent Publication No. 10-2103941.

This application is intended to solve the problems of the prior art described above, and is a semantic method that estimates the location of a moving object through matching between a pre-constructed 3D point cloud map and a semantic segmentation image processed from images acquired in real time during the moving object's driving process. The purpose is to provide a location estimation device and method through holistic matching based on segmented images and semantic point cloud maps.

However, the technical challenges sought to be achieved by the embodiments of the present application are not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-mentioned technical task, the location estimation method through holistic matching based on semantic segmented image and semantic point cloud map according to an embodiment of the present application is used in the target space, which is the surrounding space for a predetermined moving object. receiving an image input, generating a semantic segmentation image corresponding to the image input based on a first artificial intelligence model previously learned to perform object-based semantic segmentation on a predetermined input image, and It may include the step of estimating location information of the moving object by matching the constructed semantic point cloud map and the semantic segmented image.

In addition, the semantic point cloud map includes a plurality of 3D points indicating location information of a preset type of object located in a map construction space including the target space, and an object for each of the plurality of 3D points It can be built in advance so that classification information is assigned.

In addition, the step of estimating the location information includes obtaining point clusters corresponding to each of a plurality of particles representing candidate locations of the moving object from the semantic point cloud map, and projecting the point clusters on the semantic segmented image, respectively. A step of: first object classification information allocated to each of a plurality of 3D points included in the point cluster and second object classification information of the semantic segmentation image allocated to a point where each of the plurality of 3D points is projected. It may include calculating a matching score for each of the plurality of particles according to similarity between the particles, and updating a weight for each of the plurality of particles based on the matching score.

Additionally, the location information of the moving object may be derived based on the candidate location and the updated weight corresponding to each of the plurality of particles.

In addition, the plurality of particles may be particles whose positions according to changes in state information predicted based on inertial sensor information of the moving object are predicted to be within a preset distance from the moving object based on satellite navigation information of the moving object. there is.

Additionally, the state information may include coordinate information, orientation information, and height information for each of the plurality of particles.

In addition, the step of acquiring the point cluster may include extracting a plurality of 3D points included in a region of interest set by assuming that the moving object is located at each of the candidate locations from the semantic point cloud map. there is.

Additionally, acquiring the point cluster may include applying random sampling to a plurality of 3D points included in the region of interest.

Additionally, the moving object may be a vehicle.

Additionally, the preset type of object may include at least one of a lane, a traffic sign, a traffic light, a fence structure, a lighting structure, and a building.

In addition, the location estimation method through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application includes the step of constructing the semantic point cloud map before receiving the image input. can do.

In addition, the building step includes collecting a point cloud including the plurality of three-dimensional points and using a second artificial intelligence model previously trained to perform object-based semantic segmentation on a predetermined input point cloud. It may include generating a semantic point cloud corresponding to the collected point cloud based on the collected point cloud.

Meanwhile, a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application includes an image acquisition unit that receives an image input for a target space, which is the surrounding space for a predetermined moving object; An image processing unit that generates a semantic segmentation image corresponding to the image input based on a first artificial intelligence model learned in advance to perform object-based semantic segmentation on a predetermined input image, and a pre-built semantic point cloud map; It may include a matching unit that estimates location information of the moving object by matching the semantic segmented image.

In addition, the matching unit includes a particle information acquisition unit that acquires a point cluster corresponding to each of a plurality of particles representing the candidate location of the moving object from the semantic point cloud map, and a projection unit that projects the point clusters on the semantic segmented image, respectively. Part, between first object classification information assigned to each of the plurality of 3D points included in the point cluster and second object classification information of the semantic segmentation image assigned to the point where each of the plurality of 3D points is projected. It may include a matching score calculating unit that calculates a matching score according to similarity for each of the plurality of particles, and a weight updating unit that updates a weight for each of the plurality of particles based on the matching score.

In addition, a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application collects a point cloud including the plurality of three-dimensional points, and stores a predetermined input point cloud. It may include a map construction unit for constructing the semantic point cloud map by generating a semantic point cloud corresponding to the collected point cloud based on a second artificial intelligence model previously learned to perform object-based semantic segmentation. there is.

The above-described means of solving the problem are merely illustrative and should not be construed as intended to limit the present application. In addition to the exemplary embodiments described above, additional embodiments may be present in the drawings and detailed description of the invention.

According to the above-described means of solving the problem of the present application, a semantic segmented image and semantics that estimate the location of a moving object through matching between a pre-constructed 3D point cloud map and a semantic segmented image processed from an image acquired in real time during the driving process of the moving object A location estimation device and method can be provided through point cloud map-based holistic matching.

According to the above-described means of solving the problem of this application, matching-based location estimation can be performed even when the lidar sensor, which is an expensive equipment, is not equipped in the vehicle, providing a location estimation algorithm that is cheaper and has a higher realistic applicability. You can.

According to the above-described means of solving the problem of the present application, landmark information identified in the image of the target space, which is the surrounding space of the moving object, and semantic information reflected in the semantic point cloud map can be fused in real time.

However, the effects that can be obtained herein are not limited to the effects described above, and other effects may exist.

1 is a schematic configuration diagram of a moving object location estimation system including a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.
Figures 2 and 3 are conceptual diagrams to explain the position estimation process of a moving object performed by a position estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.
Figure 4 is a diagram for explaining a semantic point cloud map.
Figure 5 is a conceptual diagram for explaining a particle filter.
Figure 6 is a conceptual diagram for explaining random sampling.
Figure 7a is a diagram illustrating an example semantic segmentation image generated based on the first artificial intelligence model.
FIG. 7B is a diagram illustrating an image projected by a point cluster corresponding to a predetermined particle onto a semantic segmentation image.
Figure 7c is a diagram illustrating a matching score calculated in response to a predetermined particle.
Figure 8 is a schematic configuration diagram of a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.
Figure 9 is a detailed configuration diagram of a matching unit of a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.
Figure 10 is an operation flowchart of a location estimation method through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.

Below, with reference to the attached drawings, embodiments of the present application will be described in detail so that those skilled in the art can easily implement them. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In order to clearly explain the present application in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout this specification, when a part is said to be “connected” to another part, this means not only “directly connected” but also “electrically connected” or “indirectly connected” with another element in between. "Includes cases where it is.

Throughout this specification, when a member is said to be located “on”, “above”, “at the top”, “below”, “at the bottom”, or “at the bottom” of another member, this means that a member is located on another member. This includes not only cases where they are in contact, but also cases where another member exists between two members.

Throughout the specification of the present application, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

This application relates to an apparatus and method for location estimation through holistic matching based on semantic segmented images and semantic point cloud maps.

1 is a schematic configuration diagram of a moving object location estimation system including a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.

Referring to FIG. 1, the moving object position estimation system 10 is a position estimation device 100 (hereinafter referred to as 'position estimation device') through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application. (referred to as '100)'), and may include a camera module 11 of the moving object 1 and a map database 300.

The location estimation device 100, the camera module 11 of the mobile object 1, and the map database 300 may communicate with each other through the network 20. The network 20 refers to a connection structure that allows information exchange between nodes such as terminals and servers. Examples of such networks 20 include the 3rd Generation Partnership Project (3GPP) network, Long Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), wifi network, Bluetooth network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc., but are not limited thereto.

In the description of the embodiment of the present application, the map database 300 may be a database that stores a semantic point cloud map 1000 previously constructed for the target space, which is the surrounding space for the moving object 1. As an example, the location estimation device 100 first determines the approximate location information of the moving object 1 using satellite navigation information such as GNSS information and GPS information, and then determines the target space within a preset range from the location. It may be set and a pre-constructed semantic point cloud map 1000 for the set target space may be loaded from the map database 300, but the method is not limited to this.

Additionally, in the description of the embodiments of the present application, the mobile object 1 may be a vehicle traveling on a road, etc., but is not limited thereto. According to an embodiment of the present application, the mobile object 1 may be a robot device (not shown) that moves in an indoor space or an outdoor space. As another example, the mobile object 1 may refer to a user terminal (not shown) whose location information changes as it moves while being carried by the user. In other words, the moving object 1 disclosed herein may include various types of objects that can move in the target space in which the semantic point cloud map 1000 is constructed. Hereinafter, for convenience of explanation, an implementation example of the present application will be described assuming that the mobile object 1 is a vehicle traveling in a road environment.

Figures 2 and 3 are conceptual diagrams to explain the position estimation process of a moving object performed by a position estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.

Referring to FIGS. 2 and 3 , the position estimation device 100 may acquire (receive) an image input for the target space, which is the surrounding space for the moving object 1. For reference, the image including the road area along which the mobile object 1 shown in FIGS. 2 and 3 travels may be an exemplary representation of the above-described image input.

According to an embodiment of the present application, the mobile object 1 is equipped with a camera module 11 to acquire an image of the space around the mobile object 1 in response to the movement (driving) of the mobile object 1, and a position estimation device 100 ) may be receiving (acquiring) an image input of the target space from the camera module 11, but is not limited to this. As another example, the position estimation device 100 may detect the target space in which the moving object 1 moves from a separate photographing device (e.g., CCTV module, etc.) provided for the target space to capture the target space in which the moving object 1 moves. It may be receiving video input from a captured video.

In addition, referring to FIGS. 2 and 3, the location estimation device 100 corresponds to an image input obtained based on a first artificial intelligence model previously learned to perform object-based semantic segmentation on a certain input image. A semantic segmentation image can be created ('Landmark Perception' in Figure 2). In other words, the first artificial intelligence model of the location estimation device 100 may be an artificial intelligence model trained in advance to perform 2D semantic segmentation on image input in the form of a 2-dimensional image. .

More specifically, the first artificial intelligence model distinguishes each of a plurality of classes selected to correspond to a plurality of object types that can affect the driving of the moving object 1 from the image input and assigns them to each pixel of the image input. The corresponding class can be labeled. Meanwhile, in the description of the embodiments of the present application, the term class for distinguishing each object type may be referred to differently as a landmark, etc.

In relation to this, in the conventional semantic segmentation algorithm, RangeNet++, etc., it is learned using an existing data set (for example, SemanticKITTI, etc.) in which the type of class corresponding to the object type to be classified is predetermined as learning data. Existing data sets typically contain multiple classes, ranging from 20 to 28.

Accordingly, in the case of a semantic segmentation model that is learned simply by using an existing data set, even objects that do not significantly affect the driving of the moving object 1 may be classified into separate classes (e.g. Vegetation, Terrain separately, etc.).

In order to solve this problem, the position estimation device 100 disclosed herein selects only the core object classes (core classes) that are predicted to have an impact on the running of the moving object 1 and performs semantic segmentation. can be applied.

By way of example, the core object classifications considered by the location estimation device 100 include traffic lights, lane markings, fence structures, pillar structures, traffic signs, adjacent driving vehicles (Cars), Sidewalks, buildings, etc. may be included, and for example, the total number of classes may be kept to 10 or less to ensure high inference speed.

In addition, the position estimation device 100 determines the class corresponding to each pixel of the image input based on the above-described first artificial intelligence model, and preset a color for the determined class (in other words, a preset color for each class) color) can be displayed on each video input. Accordingly, as shown in FIGS. 2 and 7A, the semantic segmentation image is converted to display colors corresponding to classes classified by pixel, and the area corresponding to a certain object type is colored so that it is visually distinguished. You can.

In addition, referring to FIGS. 2 and 3, the location estimation device 100 matches a pre-built semantic point cloud map (1000, 'SPCM') with a semantic segmentation image generated using the first artificial intelligence model to identify the moving object. The location information in (1) can be estimated.

Figure 4 is a diagram for explaining a semantic point cloud map.

Referring to FIG. 4, in the description of the embodiment of the present application, the semantic point cloud map 1000 includes a plurality of three-dimensional points representing location information of a preset type of object located in the map construction space including the target space. However, it may be established in advance so that object classification information for the corresponding object is assigned to each of a plurality of 3D points representing the location information of each object.

Specifically, the position estimation device 100 collects a point cloud including a plurality of three-dimensional points (see (a) of FIG. 4) and performs object-based semantic segmentation on a predetermined input point cloud. A semantic point cloud map can be created in advance by generating a semantic point cloud corresponding to the collected point cloud based on a pre-trained second artificial intelligence model (see (b) of FIG. 4). Illustratively, a point cloud including a plurality of 3D points may be collected by a mobile mapping system, but is not limited thereto.

As another example, the location estimation device 100 may receive local map information corresponding to the target space from a map database 300 previously built to store the semantic point cloud map 1000.

Hereinafter, a process in which the position estimation device 100 estimates the position information of the moving object 1 by matching the semantic point cloud map 1000 and the semantic segmented image based on a particle filter will be described.

Figure 5 is a conceptual diagram for explaining a particle filter.

Referring to FIG. 5 , the position estimation device 100 may obtain a point cluster corresponding to each of a plurality of particles representing the candidate location of the moving object 1 from the semantic point cloud map 1000 .

In this regard, the particle filter randomly places particles that probabilistically generate predicted values for the posture or position of the moving object (1) on a previously constructed map, and then obtains repetitive information to determine the actual location of the moving object (1). It is a position estimation algorithm that operates to converge the estimated position of a moving object (1) with a high probability of movement. It can be applied to models using non-linear formulas and has the advantage of being fast in calculation speed and easy to implement. Since these particle filter matters are self-evident to those skilled in the art, a more detailed description will be omitted.

Specifically, the location estimation device 100 may obtain a point cluster corresponding to each of a plurality of particles representing the candidate location of the moving object 1 from the semantic point cloud map 1000. More specifically, when assuming that the moving object 1 is actually located at each candidate position of the moving object 1 corresponding to each particle, the position estimation device 100 determines the area of interest of the moving object 1 at that location. A point cluster including a plurality of 3D points within the semantic point cloud map 1000 may be extracted from the semantic point cloud map 1000 .

Meanwhile, with regard to the 'area of interest', according to an embodiment of the present application, the position estimation device 100 determines a reference distance set based on perceived importance in the surrounding space of the moving object 1, and determines the reference distance of the moving object 1. An area within a reference distance from the location or candidate location (for example, an area within a set reference distance from a specific reference point inside the moving object 1) may be set as the area of interest.

In addition, according to an embodiment of the present application, the area of interest is an area that corresponds to the traveling direction of the mobile object 1 or the shooting direction of the camera module 11 provided in the mobile object 1 among the areas within the reference distance from the mobile object 1. It can be set to . Here, the shooting direction of the camera module may be referred to differently as the field of view of the camera module 11, Field of View (FoV), etc.

As an example to aid understanding, the standard distance set for setting the area of interest may be 50m, but it is not limited to this, and includes the type of mobile object 1, the size of the road on which the mobile object 1 is traveling, and the size of the road on which the mobile object 1 is running. The reference distance may be set differently based on driving position information, etc., and may change even while the moving object 1 is traveling. Specifically, when the moving object 1 is driving forward, the front area, the lateral area, etc. of the moving object 1 may be included in the region of interest, and the rear area of the moving object 1 may not be included in the region of interest. On the contrary, when the mobile object 1 is driving backwards, the rear area, the lateral area, etc. of the mobile object 1 may be included in the region of interest, and the front area of the mobile object 1 may not be included in the region of interest.

Additionally, according to an embodiment of the present application, the reference distance value for setting the area of interest when the moving object 1 moves forward may be set to be larger than the reference distance when the moving object 1 moves backward. This may be because when the moving object 1 moves forward, the traveling speed is generally faster than when the moving object 1 moves backward, so object identification over a wider range is required. In this regard, according to an embodiment of the present application, the reference distance value for setting the area of interest may be determined based on the traveling speed information of the moving object 1. For example, when the traveling speed of the moving object 1 is high, the reference distance value for setting the area of interest may be set larger than when the traveling speed is relatively slow.

In addition, referring to FIG. 5, the position estimation device 100 determines that the position change according to the change in the state information of the particle predicted based on the inertial sensor information of the moving object 1 is satellite navigation information (GPS information) of the moving object 1. , GNSS information, etc.), particles predicted to be within a preset distance (δ _GPS in FIG. 5 ) from the moving object 1 may be selected to selectively obtain point clusters corresponding to each particle. Here, the particle state information may include coordinate information, orientation information, and height information for each of a plurality of particles.

As an example, inertial sensor information for predicting changes in the state information of each particle may be acquired by an inertial measurement sensor unit 12 provided on the moving object 1. Meanwhile, the position estimation device 100 removes the weight assigned to particles predicted to be located farther than a preset distance from the moving object 1 among a plurality of particles each disposed at a random location on the map (in other words, (update to 0) (see (b) of FIG. 5). For reference, the process of selectively selecting some of these plural particles may be referred to as a particle validation (validation check) process, etc., referring to FIGS. 2 and 5 .

Figure 6 is a conceptual diagram for explaining random sampling.

Referring to FIG. 6, the position estimation device 100 applies random sampling to a plurality of 3D points included in the region of interest to obtain a plurality of 3D points projected onto the semantic segmentation image as described later. Preprocessing can be performed to reduce the amount of computation by reducing the number. More specifically, Figure 6(a) shows the state before random sampling is applied to a plurality of three-dimensional points included in the region of interest, and Figure 6(b) shows the state after random sampling is applied and removed by random sampling. It represents a plurality of three-dimensional points that have not been formed.

Additionally, the position estimation device 100 may project each of the acquired point clusters onto the semantic segmentation image. According to an embodiment of the present application, the position estimation device 100 performs 2-dimensional semantic segmentation of each 3-dimensional point in a point cluster whose position is defined in 3-dimensional space based on a preset calibration parameter. Projection into a semantic segmented image can be performed by determining the corresponding point on the image.

In relation to this, FIG. 7A is a diagram illustrating a semantic segmentation image generated based on the first artificial intelligence model, and FIG. 7B is an image in which a point cluster corresponding to a predetermined particle is projected onto the semantic segmentation image. This is a drawing shown in detail.

In addition, the position estimation device 100 includes first object classification information assigned to each of a plurality of 3D points included in the point cluster corresponding to each particle and semantics assigned to the point where each of the plurality of 3D points is projected. A matching score according to the similarity between the second object classification information of the divided image can be calculated for each plurality of particles.

Additionally, according to an embodiment of the present application, the similarity between object classification information may be calculated individually for each of the plurality of classes described above. For example, the location estimation device 100 allocates object classification information to each object of a preset type (class) including at least one of a lane, a traffic sign, a traffic light, a fence structure, a lighting structure, and a building, and provides semantic As a result of matching the point cloud map and the semantic segmentation image, the matching score is calculated relatively higher as the 3D point is projected to the point where the same class as the class semantically assigned to the given 3D point is assigned on the semantic segmentation image. It can be.

For example, in the case of the first particle (Particle 1) according to the SPCM Projection shown in FIG. 3, object classification information (second object classification information) semantically divided on the semantic segmentation image compared to the second particle (Particle 2) and the pre-assigned object classification information (first object classification information) for each of the projected 3D points shows a relatively similar pattern (the positions on the image between the projected points and pixels to which the same object classification information is assigned are relatively (close ones) can be checked. Accordingly, the candidate position corresponding to the first particle is evaluated as being relatively closer to the actual position of the moving object (1) compared to the candidate position corresponding to the second particle, and the matching score calculated in response to the first particle is increased to a high value. can be derived.

Accordingly, the position estimation device 100 can estimate the actual position information of the moving object 1 by predicting that the candidate position according to the particle for which the matching score is calculated to be relatively high will be close to the actual position of the moving object 1. You can. More specifically, the position estimation device 100 may update the weight for each of the plurality of particles based on the matching score calculated for each of the plurality of particles. To aid understanding, as an example, as shown in (b) of FIG. 5, in the initial state, a plurality of particles are all assigned weights with equal values (for example, 0.083, referring to (b) of FIG. 5). After the matching score is calculated through the above-described process, the weight may be updated so that the weight for the particle for which the matching score is calculated to be high is assigned a high value.

Meanwhile, according to an embodiment of the present application, the matching score for each of a plurality of particles may be calculated based on an Intersection over Union (IoU) metric, but is not limited thereto, and according to an implementation of the present application, the matching score may be calculated based on the Intersection over Union (IoU) metric. ), it may be calculated based on metrics for performance evaluation, such as recall.

In this regard, according to an embodiment of the present application, the location estimation device 100 includes, for each object type (class), first object classification information corresponding to the class among a plurality of three dimensions projected on the semantic segmentation image. The area of overlap between the first area, which is the area occupied by the assigned 3D point, and the second area, which is the area where the second object classification information corresponding to the class is assigned on the semantic segmentation image, is divided into the first area and the second area. The matching score can be calculated based on the IoU (Intersection over Union) metric by dividing the two areas by the area of the union (Area of Union), but the matching score is not limited to this.

Figure 7c is a diagram illustrating a matching score calculated in response to a predetermined particle.

Referring to FIG. 7C, the location estimation device 100 may individually calculate a matching score for each object type (class) identified in the area of interest. By way of example, Figure 7c shows the information for each class, such as traffic light, traffic sign, pole, fence, lane marking, and building. Matching scores can be derived individually. Additionally, the position estimation device 100 may update the weight of each of the plurality of particles by combining the matching scores derived for each class (eg, sum operation, etc.).

Figure 8 is a schematic configuration diagram of a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.

Referring to FIG. 8 , the location estimation device 100 may include a map construction unit 110, an image acquisition unit 120, an image processing unit 130, and a matching unit 140.

The map construction unit 110 may collect a point cloud including a plurality of 3D points.

In addition, the map construction unit 110 generates a semantic point cloud corresponding to the collected point cloud based on a second artificial intelligence model trained in advance to perform object-based semantic segmentation on a predetermined input point cloud. A semantic point cloud map (1000) can be constructed.

The image acquisition unit 120 may receive an image input for the target space, which is the surrounding space for a certain moving object 1.

The image processing unit 130 may generate a semantic segmentation image corresponding to the received image input based on a first artificial intelligence model previously learned to perform object-based semantic segmentation on a certain input image.

The matching unit 140 may estimate the location information of the moving object 1 by matching the pre-built semantic point cloud map 1000 and the generated semantic segmented image.

Figure 9 is a detailed configuration diagram of a matching unit of a location estimation device through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.

Referring to FIG. 9, the matching unit 140 may include a particle information acquisition unit 141, a projection unit 142, a matching score calculation unit 143, and a weight update unit 144.

The particle information acquisition unit 141 may obtain a point cluster corresponding to each of a plurality of particles representing the candidate location of the moving object 1 from the semantic point cloud map 1000.

The projection unit 142 may project each of the acquired point clusters onto the semantic segmentation image generated by the image processing unit 130.

The matching score calculation unit 143 stores first object classification information assigned to each of the plurality of 3D points included in the point cluster corresponding to each particle and each of the plurality of 3D points of the point cluster on the semantic segmentation image. A matching score according to the similarity between the second object classification information of the semantic segmentation image assigned to the projected point can be calculated for each plurality of particles.

The weight update unit 144 may update the weight for each of the plurality of particles based on the calculated matching score.

In addition, the matching unit 140 derives (e.g., probabilistically determines) the position information of the moving object 1 based on the candidate positions corresponding to each of the plurality of particles and the weight updated by the weight updating unit 144. )can do.

Below, we will briefly look at the operation flow of the present application based on the details described above.

Figure 10 is an operation flowchart of a location estimation method through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application.

The location estimation method through holistic matching based on the semantic segmented image and semantic point cloud map shown in FIG. 10 can be performed by the location estimation device 100 described above. Therefore, even if the content is omitted below, the description of the location estimation device 100 can be equally applied to the description of the location estimation method through holistic matching based on semantic segmented images and semantic point cloud maps.

Referring to FIG. 10, in step S11, the map construction unit 110 corresponds to a point cloud collected based on a second artificial intelligence model previously learned to perform object-based semantic segmentation on a predetermined input point cloud. A semantic point cloud map (1000) can be constructed by creating a semantic point cloud.

Next, in step S12, the image acquisition unit 120 may receive an image input for the target space, which is the surrounding space for a certain moving object 1.

Next, in step S13, the image processing unit 130 generates semantic information corresponding to the image input received in step S12 based on a first artificial intelligence model previously learned to perform object-based semantic segmentation on a predetermined input image. A split image can be created.

Next, in step S14, the matching unit 140 may estimate the location information of the moving object 1 by matching the pre-built semantic point cloud map 1000 with the semantic segmented image generated through step S13.

Specifically, in step S14, the particle information acquisition unit 141 may obtain a point cluster corresponding to each of a plurality of particles representing the candidate location of the moving object 1 from the semantic point cloud map 1000.

Additionally, in step S14, the projection unit 142 may project each of the acquired point clusters onto the semantic segmentation image generated by the image processing unit 130.

In addition, in step S14, the matching score calculation unit 143 determines that the first object classification information assigned to each of the plurality of 3D points included in the point cluster corresponding to each particle and each of the plurality of 3D points of the point cluster are A matching score according to the similarity between the second object classification information of the semantic segmented image assigned to the point projected on the semantic segmented image may be calculated for each plurality of particles.

Additionally, in step S14, the weight update unit 144 may update the weight for each of the plurality of particles based on the calculated matching score.

Additionally, in step S14, the matching unit 140 may derive the position information of the moving object 1 based on the candidate positions corresponding to each of the plurality of particles and the weight updated by the weight updating unit 144.

In the above description, steps S11 to S14 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Additionally, some steps may be omitted or the order between steps may be changed as needed.

The location estimation method through holistic matching based on a semantic segmented image and a semantic point cloud map according to an embodiment of the present application can be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. . The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and constructed for the present invention or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

Additionally, the location estimation method through holistic matching based on the above-described semantic segmented image and semantic point cloud map may also be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The description of the present application described above is for illustrative purposes, and those skilled in the art will understand that the present application can be easily modified into other specific forms without changing its technical idea or essential features. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

10: Moving object position estimation system
100: Location estimation device through holistic matching based on semantic segmented image and semantic point cloud map
110: Map construction unit
120: Image acquisition unit
130: Image processing unit
140: Matching unit
141: Particle information acquisition unit
142: Projection unit
143: Matching score calculation unit
144: Weight update unit
300: Map database
20: Network
1: Moving body
11: Camera module
1000: Semantic point cloud map

Claims (15)

A location estimation method through holistic matching based on semantic segmented images and semantic point cloud maps,
Receiving an image input for a target space, which is a surrounding space for a predetermined moving object;
generating a semantic segmentation image corresponding to the image input based on a first artificial intelligence model previously trained to perform object-based semantic segmentation on the input image; and
estimating location information of the moving object by matching the semantic segmented image with a pre-built semantic point cloud map;
Including,
The step of estimating the location information is,
Obtaining a point cluster corresponding to each of a plurality of particles representing a candidate location of the moving object from the semantic point cloud map;
Projecting each of the point clusters onto the semantic segmentation image;
The similarity between the first object classification information assigned to each of the plurality of 3D points included in the point cluster and the second object classification information of the semantic segmentation image assigned to the point where each of the plurality of 3D points is projected calculating a corresponding matching score for each of the plurality of particles; and
Updating a weight for each of the plurality of particles based on the matching score,
Including,
The plurality of particles are,
Characterized in that the particle is predicted to have a position according to a change in state information predicted based on inertial sensor information of the moving object within a preset reference distance from the moving object based on satellite navigation information of the moving object,
The step of acquiring the point cluster is,
Updating the weight for a particle predicted to be located further than the reference distance from the moving object among a plurality of particles each disposed at a random location on the semantic point cloud map,
The step of acquiring the point cluster is,
Extracting a plurality of 3D points included in a region of interest set by assuming that the moving object is located at each of the candidate locations from the semantic point cloud map,
A location estimation method comprising: According to paragraph 1,
The semantic point cloud map is,
Contains a plurality of 3-dimensional points representing location information of a preset type of object located in a map construction space including the target space, and is pre-built so that object classification information is assigned to each of the plurality of 3-dimensional points In,position estimation method. delete According to paragraph 1,
The location information of the moving object is,
A position estimation method that is derived based on the candidate position and the updated weight corresponding to each of the plurality of particles. delete According to paragraph 1,
The status information is,
A position estimation method including coordinate information, orientation information, and height information for each of the plurality of particles. delete According to paragraph 1,
The step of acquiring the point cluster is,
Applying random sampling to a plurality of 3D points included in the region of interest,
A location estimation method further comprising: According to paragraph 2,
The moving object is a vehicle,
Objects of the preset type are:
A location estimation method comprising at least one of lanes, traffic signs, traffic lights, fence structures, lighting structures, and buildings. According to paragraph 2,
Before receiving the video input,
Constructing the semantic point cloud map,
It further includes,
The construction step is,
collecting a point cloud including the plurality of 3D points; and
Generating a semantic point cloud corresponding to the collected point cloud based on a second artificial intelligence model trained in advance to perform object-based semantic segmentation on an input predetermined point cloud,
A location estimation method comprising: A location estimation device through holistic matching based on semantic segmented images and semantic point cloud maps,
an image acquisition unit that receives an image input for a target space, which is a surrounding space for a predetermined moving object;
an image processing unit that generates a semantic segmentation image corresponding to the image input based on a first artificial intelligence model previously learned to perform object-based semantic segmentation on the input image; and
A matching unit that matches a pre-built semantic point cloud map and the semantic segmented image to estimate location information of the moving object;
Including,
The matching unit,
a particle information acquisition unit that acquires a point cluster corresponding to each of a plurality of particles representing candidate locations of the moving object from the semantic point cloud map;
a projection unit respectively projecting the point clusters onto the semantic segmentation image;
The similarity between the first object classification information assigned to each of the plurality of 3D points included in the point cluster and the second object classification information of the semantic segmentation image assigned to the point where each of the plurality of 3D points is projected a matching score calculation unit that calculates a matching score for each of the plurality of particles; and
A weight update unit that updates a weight for each of the plurality of particles based on the matching score,
Including,
The plurality of particles are,
Characterized in that the particle is predicted to have a position according to a change in state information predicted based on inertial sensor information of the moving object within a preset reference distance from the moving object based on satellite navigation information of the moving object,
The particle information acquisition unit,
Updating the weight for a particle predicted to be located further than the reference distance from the moving object among a plurality of particles each disposed at a random location on the semantic point cloud map,
The particle information acquisition unit,
A position estimation device that extracts a plurality of 3D points included in a region of interest set by assuming that the moving object is located at each of the candidate locations from the semantic point cloud map. According to clause 11,
The semantic point cloud map is,
Contains a plurality of 3-dimensional points representing location information of a preset type of object located in a map construction space including the target space, and is pre-built so that object classification information is assigned to each of the plurality of 3-dimensional points In,position estimation device. delete delete According to clause 12,
Collects a point cloud including the plurality of three-dimensional points, and corresponds to the collected point cloud based on a second artificial intelligence model that has been pre-trained to perform object-based semantic segmentation on a predetermined input point cloud. A map construction unit for constructing the semantic point cloud map by generating a semantic point cloud,
A location estimation device further comprising:

Images