KR20230070868A - Apparatus and method for estimating location based on holistic matching using semantic segmentation image and semantic point cloud map - Google Patents

Abstract

Disclosed are an apparatus and a method for estimating a location based on holistic matching using a semantic segmentation image and a semantic point cloud map. According to an embodiment of the present invention, a method for estimating a location based on holistic matching using a semantic segmentation image and a semantic point cloud map comprises the steps of: receiving an image input for a target space, which is a surrounding space for a predetermined moving object; generating a semantic segmentation image corresponding to the image input based on a first artificial intelligence model trained in advance so as to perform object-based semantic segmentation on the predetermined input image; and estimating location information of the moving object by matching the semantic segmented image with a pre-built semantic point cloud map. The present invention can provide a position estimation algorithm with high realistic applicability.

Description

Apparatus and method for estimating location through holistic matching based on semantic segmentation image and semantic point cloud map

The present disclosure relates to an apparatus and method for estimating a location through holistic matching based on a semantic segmentation image and a semantic point cloud map.

The term Smart Car is a convergence of electrical, electronic, and information and communication technologies with traditional machine-oriented automotive technology. It has been continuously developed with a focus on aspects, and these smart vehicle-related technologies are specifically related to safety and safety, such as Advanced Driver Assistance System (ADAS), Autonomous Vehicle, and Cooperative Safety System. It can be classified into related fields and convenience-related fields such as connected cars.

In particular, an intelligent car equipped with Advanced Driver Assistance Systems (ADAS) or Autonomous Vehicle (Autonomous Vehicle) is equipped with various sensors (e.g., cameras, GPS, radar, LiDAR, etc.) It should be possible to secure the safety of occupants, pedestrians and surrounding vehicles and improve driving convenience by monitoring the situation around the vehicle in real time with high accuracy through the acquired sensing data.

In particular, LiDAR is a sensor that recognizes the road environment around the car as a three-dimensional shape by determining the shape of the target object by measuring the distance to a specific target with a laser. By providing precise measurement performance, there is an advantage that can be used in various vehicle-linked recognition systems such as obstacle recognition and dynamic object tracking.

However, in the case of a map-based position estimation algorithm using a conventional 3D lidar sensor, high accuracy at the centimeter level is guaranteed, but it is applicable only when an expensive lidar sensor is mounted on the vehicle, so it is not suitable for mass production vehicles. There are limitations that are difficult to apply.

The background technology of the present application is disclosed in Korean Patent Registration No. 10-2103941.

The present invention is intended to solve the above-mentioned problems of the prior art, and semantics for estimating the location of a moving object through matching between a pre-constructed 3D point cloud map and a semantic segmentation image obtained by processing an image acquired in real time while the moving object is traveling. An object of the present invention is to provide an apparatus and method for estimating location through holistic matching based on segmented images and semantic point cloud maps.

However, the technical problem to be achieved by the embodiments of the present application is not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, a position estimation method through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present invention is performed in a target space, which is a space around a predetermined moving object. Receiving an image input for the input image, generating a semantic segmentation image corresponding to the image input based on a pre-learned first artificial intelligence model to perform object-based semantic segmentation on a predetermined image input, and The method may include estimating location information of the moving object by matching the constructed semantic point cloud map and the semantic segmentation image.

In addition, the semantic point cloud map includes a plurality of 3D points representing location information of a preset type of object located in a map construction space including the target space, and an object for each of the plurality of 3D points. Classification information may be pre-established to be assigned.

In addition, the estimating the location information may include obtaining a point cluster corresponding to each of a plurality of particles indicating the candidate location of the moving object from the semantic point cloud map, and projecting the point cluster onto the semantic segmentation image, respectively. first object classification information assigned to each of a plurality of 3D points included in the point cluster and second object classification information of the semantic segmentation image assigned to a point where each of the plurality of 3D points is projected The method may include calculating a matching score for each of the plurality of particles according to a degree of similarity between particles and updating a weight for each of the plurality of particles based on the matching score.

Also, the location information of the moving object may be derived based on the candidate location corresponding to each of the plurality of particles and the updated weight.

In addition, the plurality of particles may be particles predicted to have a position according to a change in state information predicted based on inertial sensor information of the moving object within a preset distance from the moving object based on satellite navigation information of the moving object. there is.

In addition, the state information may include coordinate information, orientation information, and height information of each of the plurality of particles.

The obtaining of the point cluster may include extracting a plurality of 3D points included in a region of interest set by assuming that the moving object is located at each of the candidate positions from the semantic point cloud map. there is.

Also, the obtaining of the point cluster may include applying random sampling to a plurality of 3D points included in the ROI.

Also, the moving body may be a vehicle.

Also, the preset type of object may include at least one of a lane, a traffic sign, a traffic light, a fence structure, a lighting structure, and a building.

In addition, the location estimation method through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present application includes constructing the semantic point cloud map before receiving the image input. can do.

In addition, the constructing step may include the collecting of the point cloud including the plurality of 3D points and the second artificial intelligence model pre-learned to perform object-based semantic segmentation on the input predetermined point cloud. and generating a semantic point cloud corresponding to the collected point cloud based on the method.

Meanwhile, an apparatus for estimating a position through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present application includes an image acquisition unit receiving an image input for a target space, which is a space around a predetermined moving object, An image processing unit that generates a semantic segmentation image corresponding to an image input based on a first artificial intelligence model learned in advance to perform object-based semantic segmentation on a predetermined input image and a pre-constructed semantic point cloud map A matching unit for estimating location information of the moving object by matching the semantic segmentation image may be included.

In addition, the matching unit may include a particle information obtaining unit acquiring point clusters corresponding to each of a plurality of particles indicating candidate positions of the moving object from the semantic point cloud map, and a projection projecting the point clusters onto the semantic segmentation image, respectively. between the first object classification information allocated to each of the plurality of 3D points included in the point cluster and the second object classification information of the semantic segmentation image allocated to the point where each of the plurality of 3D points is projected It may include a matching score calculation unit that calculates a matching score according to similarity for each of the plurality of particles, and a weight update unit that updates a weight for each of the plurality of particles based on the matching score.

In addition, an apparatus for estimating a position through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present application collects a point cloud including the plurality of 3D points and inputs a predetermined point cloud. It may include a map building unit for constructing the semantic point cloud map by generating a semantic point cloud corresponding to the collected point cloud based on a second artificial intelligence model learned in advance to perform object-based semantic segmentation for there is.

The above-described problem solving means are merely exemplary and should not be construed as intended to limit the present disclosure. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-mentioned problem solving means of the present application, a semantic segmentation image and semantic segmentation image for estimating the position of a moving object through matching between a pre-constructed 3D point cloud map and a semantic segmentation image obtained by processing an image obtained in real time during the driving process of the moving body. It is possible to provide a location estimation device and method through holistic matching based on a point cloud map.

According to the above-described problem solving means of the present application, even when an expensive lidar sensor is not provided in a vehicle, it is possible to perform matching-based position estimation, thereby providing a position estimation algorithm that is cheaper and has a high possibility of practical application. can

According to the above-described problem solving means of the present application, landmark information identified in an image of a target space, which is a space around a moving object, and semantic information reflected in a semantic point cloud map can be converged in real time.

However, the effects obtainable herein are not limited to the effects described above, and other effects may exist.

1 is a schematic configuration diagram of a system for estimating a location of a moving object including a location estimation device through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present invention.
2 and 3 are conceptual diagrams for explaining a location estimation process of a moving object performed by a location estimation device through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present invention.
4 is a diagram for explaining a semantic point cloud map.
5 is a conceptual diagram for explaining a particle filter.
6 is a conceptual diagram for explaining random sampling.
7A is a diagram illustrating a semantic segmentation image generated based on a first artificial intelligence model by way of example.
FIG. 7B is a diagram exemplarily illustrating an image obtained by projecting a point cluster corresponding to a predetermined particle onto a semantic segmentation image.
7C is a diagram showing a matching score calculated corresponding to a predetermined particle by way of example.
8 is a schematic configuration diagram of an apparatus for estimating a location through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present disclosure.
9 is a detailed configuration diagram of a matching unit of a location estimation device through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present disclosure.
10 is an operational flowchart for a location estimation method through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present invention.

Hereinafter, embodiments of the present application will be described in detail so that those skilled in the art can easily practice with reference to the accompanying drawings. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. And in order to clearly describe the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is said to be “connected” to another part, it is not only “directly connected”, but also “electrically connected” or “indirectly connected” with another element in between. "Including cases where

Throughout the present specification, when a member is referred to as being “on,” “above,” “on top of,” “below,” “below,” or “below” another member, this means that a member is located in relation to another member. This includes not only the case of contact but also the case of another member between the two members.

Throughout the present specification, when a certain component is said to "include", it means that it may further include other components without excluding other components unless otherwise stated.

The present disclosure relates to an apparatus and method for estimating a location through holistic matching based on a semantic segmentation image and a semantic point cloud map.

1 is a schematic configuration diagram of a system for estimating a location of a moving object including a location estimation device through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present invention.

Referring to FIG. 1, the mobile body position estimation system 10 is a position estimation device 100 through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present application (hereinafter, 'position estimation device (100)'), the camera module 11 of the mobile body 1, and the map database 300.

The location estimating device 100 , the camera module 11 of the moving object 1 , and the map database 300 may communicate with each other through the network 20 . The network 20 refers to a connection structure capable of exchanging information between nodes such as terminals and servers, and examples of such a network 20 include a 3rd Generation Partnership Project (3GPP) network and a Long LTE (LTE) network. Term Evolution (Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network) Network), wifi network, Bluetooth network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc. are included, but are not limited thereto.

In the description of the embodiment of the present application, the map database 300 may be a database that stores a pre-constructed semantic point cloud map 1000 for a target space, which is a space around the moving object 1 . Exemplarily, the location estimating device 100 primarily determines approximate location information of the moving object 1 by using satellite navigation information such as GNSS information and GPS information, and then locates a target space within a preset range from the corresponding location. It may be set and the semantic point cloud map 1000 pre-constructed for the set target space is loaded from the map database 300, but is not limited thereto.

In addition, in the description of the embodiment of the present application, the movable body 1 may be a vehicle traveling on a road or the like, but is not limited thereto. According to one embodiment of the present application, the movable body 1 may be a robot device (not shown) that moves in an indoor space or an outdoor space. As another example, the moving object 1 may refer to a user terminal (not shown) that moves while being carried by a user and whose location information is changed. In other words, the moving object 1 disclosed herein may include various types of objects movable on the target space in which the semantic point cloud map 1000 is constructed. Hereinafter, for convenience of description, an embodiment of the present disclosure will be described on the assumption that the movable body 1 is a vehicle traveling in a road environment.

2 and 3 are conceptual diagrams for explaining a location estimation process of a moving object performed by a location estimation device through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present invention.

Referring to FIGS. 2 and 3 , the position estimating device 100 may acquire (receive) an image input for a target space, which is a space around the moving object 1 . For reference, the image including the road area on which the moving object 1 travels shown in FIGS. 2 and 3 may represent the above-described image input as an example.

According to one embodiment of the present application, the mobile body 1 is equipped with a camera module 11 to acquire an image of the space around the mobile body 1 in response to the movement (traveling) of the mobile body 1, and the position estimation device 100 ) may be to receive (acquire) an image input of the target space from the camera module 11, but is not limited thereto. As another example, the position estimating device 100 is a target space in which the moving object 1 travels from a separate photographing device (eg, CCTV module, etc.) It may be to receive an image input photographed.

In addition, referring to FIGS. 2 and 3, the location estimation device 100 corresponds to an image input obtained based on a pre-learned first artificial intelligence model to perform object-based semantic segmentation on a predetermined image. It is possible to generate a semantic segmentation image ('Landmark Perception' in FIG. 2). In other words, the first artificial intelligence model of the location estimation device 100 may be an artificial intelligence model pre-learned to perform 2D semantic segmentation on an image input in the form of a 2D image. .

More specifically, the first artificial intelligence model separates each of a plurality of classes selected to correspond to a plurality of object types that may affect the driving of the moving object 1 from the image input, and assigns each pixel of the image input A corresponding class can be labeled. Meanwhile, in the description of the embodiments of the present application, the term class for classifying each object type may be differently referred to as a landmark.

In this regard, in the conventional semantic segmentation algorithm, such as RangeNet++, the type of class corresponding to the object type to be classified is pre-determined. Existing data sets typically include a plurality of classes of 20 to 28.

Accordingly, in the case of a semantic segmentation model learned by simply utilizing an existing data set, even objects that do not significantly affect the driving of the moving object 1 may be classified into separate classes (eg, Vegetation, Terrain separately, etc.).

In order to solve this problem, the position estimating device 100 disclosed herein selects only core object classifications (core classes) that are predicted to have an effect on the driving of the moving object 1 and semantic segmentation (Semantic Segmentation) can be applied.

Exemplarily, core object classifications considered by the location estimating device 100 include traffic lights, lane markings, fence structures, pillar structures, traffic signs, adjacent vehicles (cars), A sidewalk, a building, and the like may be included, and the total number of classes may be maintained to 10 or less to secure a high inference speed.

In addition, the location estimating device 100 determines a class corresponding to each pixel of the image input based on the above-described first artificial intelligence model, and a preset color for the determined class (in other words, a preset color for each class) color) can be displayed on each video input. Accordingly, as shown in FIGS. 2 and 7A, the semantic segmentation image is converted to display a color corresponding to a class classified for each pixel, and colorized so that a region corresponding to a predetermined object type is visually distinguished. can

In addition, referring to FIGS. 2 and 3, the location estimating device 100 matches a pre-built semantic point cloud map (1000, 'SPCM') with a semantic segmentation image generated using the first artificial intelligence model to move the object. The location information of (1) can be estimated.

4 is a diagram for explaining a semantic point cloud map.

Referring to FIG. 4 , in the description of an embodiment of the present application, a semantic point cloud map 1000 includes a plurality of 3D points representing location information of a preset type of object located in a map construction space including a target space. However, it may be constructed in advance so that object classification information of a corresponding object is allocated to each of a plurality of 3D points representing location information of each object.

Specifically, the location estimating device 100 collects point clouds including a plurality of 3D points (see FIG. 4(a)), and performs object-based semantic segmentation on a predetermined point cloud that is input. A semantic point cloud map may be generated in advance by generating a semantic point cloud corresponding to the collected point cloud based on the pre-learned second artificial intelligence model (see FIG. 4(b)). Illustratively, the point cloud including a plurality of 3D points may be collected by a mobile vehicle measurement system (Mobile Mapping System), but is not limited thereto.

As another example, the location estimating device 100 may receive local map information corresponding to the target space from the map database 300 previously built to store the semantic point cloud map 1000 .

Hereinafter, a process of estimating the location information of the moving object 1 by matching the semantic point cloud map 1000 and the semantic segmentation image based on a particle filter by the location estimating device 100 will be described.

5 is a conceptual diagram for explaining a particle filter.

Referring to FIG. 5 , the location estimating device 100 may obtain a point cluster corresponding to each of a plurality of particles indicating the candidate location of the moving object 1 from the semantic point cloud map 1000 .

In this regard, the particle filter randomly places particles that probabilistically generate predicted values for the position or position of the moving object 1 on a pre-built map, and then repeatedly acquires information to determine the actual location of the moving object 1. As a position estimation algorithm that operates so that the estimated position of the moving object 1 converges with particles with a high probability of falling, it can be applied to a model using a non-linear form, and has the advantage of fast operation speed and easy implementation. Since the details of the particle filter are obvious to those skilled in the art, a detailed description thereof will be omitted.

In detail, the location estimating device 100 may obtain a point cluster corresponding to each of a plurality of particles indicating the candidate location of the moving object 1 from the semantic point cloud map 1000 . More specifically, when it is assumed that the moving object 1 is actually located at each of the candidate positions of the moving object 1 corresponding to each particle, the position estimating device 100 determines the region of interest of the moving object 1 at the corresponding position. A point cluster including a plurality of 3D points in the included semantic point cloud map 1000 may be extracted from the semantic point cloud map 1000 .

Meanwhile, in relation to the 'region of interest', according to an embodiment of the present application, the position estimating device 100 determines a reference distance set based on the perceived importance in the space around the moving object 1, and An area within a reference distance from the position or candidate position (eg, an area within a set reference distance from a specific reference point inside the moving object 1) may be set as the region of interest.

In addition, according to an embodiment of the present application, the region of interest is an area corresponding to the traveling direction of the moving body 1 or the photographing direction of the camera module 11 provided in the moving body 1 among the areas within the reference distance from the moving body 1. can be set to Here, the photographing direction of the camera module may be otherwise referred to as a field of view of the camera module 11, a field of view (FoV), and the like.

As an example for better understanding, the reference distance set for setting the region of interest may be 50 m, but is not limited thereto, and the type of moving object 1, the size of the road on which the moving object 1 is running, the size of the moving object 1 The reference distance may be differently set based on driving position information and the like, and may vary even while the moving object 1 is driving. Specifically, when the moving object 1 is being driven forward, the front region and the side region of the moving object 1 may be included in the region of interest, and the rear region of the moving object 1 may not be included in the region of interest. Conversely, when the mobile body 1 is driven backward, the rear region and the side region of the mobile body 1 may be included in the ROI, and the front region of the mobile body 1 may not be included as the ROI.

Also, according to one embodiment of the present application, a reference distance value for setting a region of interest may be set to be greater than a reference distance when the moving object 1 moves forward than a reference distance when the moving object 1 moves backward. This may be because object identification for a wider range is required because when the moving object 1 moves forward, the travel speed is generally faster than when it moves backward. In this regard, according to an embodiment of the present application, a reference distance value for setting a region of interest may be determined based on driving speed information of the moving object 1 . For example, a reference distance value for setting a region of interest may be set higher when the traveling speed of the moving object 1 is higher than when the traveling speed is relatively slow.

In addition, referring to FIG. 5, the position estimating device 100 determines that the position of change according to the change in the state information of the particle predicted based on the inertial sensor information of the moving object 1 is the satellite navigation information (GPS information) of the moving object 1. , GNSS information, etc.) may select particles predicted to be within a preset distance (δ _GPS in FIG. 5 ) from the moving object 1 and selectively obtain a point cluster corresponding to each particle. Here, the particle state information may include coordinate information, orientation information, and height information of each of a plurality of particles.

Exemplarily, inertial sensor information for predicting changes in state information of each particle may be acquired by an inertial measurement sensor unit 12 or the like provided in the moving body 1 . Meanwhile, the location estimating apparatus 100 removes weights given to particles predicted to be located farther than a preset distance from the moving object 1 among a plurality of particles each disposed at an arbitrary location on the map (in other words, update to 0) (see (b) of FIG. 5). For reference, the process of selectively selecting some of the plurality of particles may be referred to as a particle validation (validation check) process or the like, referring to FIGS. 2 and 5 .

6 is a conceptual diagram for explaining random sampling.

Referring to FIG. 6 , the location estimating apparatus 100 applies random sampling to a plurality of 3D points included in the region of interest to generate a plurality of 3D points projected on a semantic segmented image as will be described later. Preprocessing can be performed to reduce the amount of computation by reducing the number. More specifically, (a) of FIG. 6 shows a state before random sampling is applied to a plurality of 3D points included in the region of interest, and (b) of FIG. 6 is removed by random sampling after random sampling is applied. It represents a plurality of three-dimensional points that have not been identified.

Also, the location estimating device 100 may project each of the acquired point clusters onto the semantic segmentation image. According to an embodiment of the present application, the location estimation device 100 is configured to perform 2-dimensional semantic segmentation of each 3-dimensional point in a point cluster whose location is defined in a 3-dimensional space based on a preset calibration parameter. Projection into a semantic segmentation image may be performed by determining a corresponding point on the image.

In this regard, FIG. 7A is a diagram exemplarily illustrating a semantic segmentation image generated based on the first artificial intelligence model, and FIG. 7B illustrates an image obtained by projecting a point cluster corresponding to a predetermined particle onto the semantic segmentation image. It is an enemy drawing.

In addition, the location estimating device 100 includes first object classification information assigned to each of a plurality of 3D points included in a point cluster corresponding to each particle and semantics assigned to a point where each of the plurality of 3D points is projected. A matching score according to the similarity between the second object classification information of the divided image may be calculated for each of a plurality of particles.

Also, according to an embodiment of the present application, the degree of similarity between object classification information may be individually calculated for each of the plurality of classes described above. For example, the location estimating device 100 allocates object classification information to each object of a preset type (class) including at least one of a lane, a traffic sign, a traffic light, a fence structure, a lighting structure, and a building, and semantic As a result of matching between the point cloud map and the semantic segmentation image, the matching score is calculated relatively higher as the 3D point is projected to the point where the same class as the class semantically assigned to the predetermined 3D point is assigned to the semantic segmentation image. It can be.

Exemplarily, in the case of the first particle (Particle 1) according to the SPCM projection shown in FIG. 3, semantically segmented object classification information (second object classification information) on the semantic segmentation image compared to the second particle (Particle 2) For each of the projected 3D points, pre-assigned object classification information (first object classification information) shows a relatively similar pattern (the position on the image between the projected point and the pixel to which the same object classification information is assigned is relatively close ones) can be checked. Accordingly, the candidate position corresponding to the first particle is evaluated to be relatively closer to the actual position of the moving object 1 compared to the candidate position corresponding to the second particle, and the matching score calculated corresponding to the first particle becomes a high value. can be derived.

Accordingly, the location estimating device 100 can estimate the actual location information of the moving object 1 by predicting that the candidate location according to the particle whose matching score is calculated to be relatively high will be close to the actual location of the moving object 1. can More specifically, the location estimating device 100 may update a weight for each of a plurality of particles based on a matching score calculated for each of a plurality of particles. For ease of understanding, as shown in (b) of FIG. 5, in the initial state, a plurality of particles all assigned weights equal to each other (eg, referring to (b) of FIG. 5, 0.083) After the matching score is calculated through the above-described process, the weight may be updated so that a high weight is assigned to a particle having a high matching score.

On the other hand, according to an embodiment of the present application, the matching score for each of a plurality of particles may be calculated based on an intersection over union (IoU) metric, but is not limited thereto, and precision ), it may be calculated based on a metric for performance evaluation such as recall.

In this regard, according to an embodiment of the present application, the location estimating device 100 includes first object classification information corresponding to a corresponding class among a plurality of three-dimensional projections on the semantic segmentation image for each object type (class). The first area, which is the area occupied by the assigned 3D points, and the second area, which is the area to which the second object classification information corresponding to the corresponding class is allocated on the semantic segmentation image, overlap the first area and the second area. A matching score may be calculated based on an intersection over union (IoU) metric by dividing the two regions by the area of union, but is not limited thereto.

7C is a diagram showing a matching score calculated corresponding to a predetermined particle by way of example.

Referring to FIG. 7C , the location estimating device 100 may individually calculate a matching score for each object type (class) identified in the region of interest. Illustratively, FIG. 7C is a diagram for each class such as a traffic light, a traffic sign, a pole structure, a fence structure, a lane marking, and a building. Matching scores can be individually derived. In addition, the location estimating apparatus 100 may update the weight of each of a plurality of particles by integrating (eg, a sum operation, etc.) the matching scores derived for each class.

8 is a schematic configuration diagram of an apparatus for estimating a location through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present disclosure.

Referring to FIG. 8 , the location estimation device 100 may include a map builder 110 , an image acquirer 120 , an image processor 130 and a matcher 140 .

The map builder 110 may collect a point cloud including a plurality of 3D points.

In addition, the map builder 110 generates a semantic point cloud corresponding to the collected point cloud based on the pre-learned second artificial intelligence model to perform object-based semantic segmentation on a predetermined point cloud input, A semantic point cloud map 1000 may be constructed.

The image acquisition unit 120 may receive an image input of a target space, which is a space surrounding a predetermined moving object 1 .

The image processing unit 130 may generate a semantic segmentation image corresponding to the received image input based on a pre-learned first artificial intelligence model to perform object-based semantic segmentation on a predetermined input image.

The matching unit 140 may estimate the location information of the moving object 1 by matching the pre-constructed semantic point cloud map 1000 with the generated semantic segmentation image.

9 is a detailed configuration diagram of a matching unit of a location estimation device through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present disclosure.

Referring to FIG. 9 , the matching unit 140 may include a particle information acquisition unit 141, a projection unit 142, a matching score calculation unit 143, and a weight update unit 144.

The particle information acquisition unit 141 may acquire point clusters corresponding to each of a plurality of particles indicating the candidate position of the moving object 1 from the semantic point cloud map 1000 .

The projection unit 142 may respectively project the acquired point clusters onto the semantic segmentation image generated by the image processing unit 130 .

The matching score calculation unit 143 stores the first object classification information assigned to each of the plurality of 3D points included in the point cluster corresponding to each particle and each of the plurality of 3D points of the corresponding point cluster on the semantic segmentation image. A matching score according to the similarity between the second object classification information of the semantic segmentation image assigned to the projected point may be calculated for each of a plurality of particles.

The weight updater 144 may update a weight for each of a plurality of particles based on the calculated matching score.

In addition, the matching unit 140 derives position information of the moving object 1 based on the candidate position corresponding to each of the plurality of particles and the weight updated by the weight updating unit 144 (eg, determined probabilistically). )can do.

Hereinafter, based on the details described above, the operation flow of the present application will be briefly reviewed.

10 is an operational flowchart for a location estimation method through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present invention.

The location estimation method through holistic matching based on the semantic segmentation image and the semantic point cloud map shown in FIG. 10 may be performed by the location estimation device 100 described above. Therefore, even if omitted below, the description of the location estimation device 100 can be equally applied to the description of the location estimation method through holistic matching based on the semantic segmentation image and the semantic point cloud map.

Referring to FIG. 10, in step S11, the map builder 110 corresponds to the point cloud collected based on the pre-learned second artificial intelligence model to perform object-based semantic segmentation on a predetermined point cloud. The semantic point cloud map 1000 may be constructed by generating a semantic point cloud that

Next, in step S12, the image acquiring unit 120 may receive an image input of a target space, which is a space surrounding the predetermined moving object 1.

Next, in step S13, the image processing unit 130 performs object-based semantic segmentation on the input predetermined image based on the pre-learned first artificial intelligence model, and semantics corresponding to the image input received in step S12. Split images can be created.

Next, in step S14, the matching unit 140 may estimate the location information of the moving object 1 by matching the pre-constructed semantic point cloud map 1000 with the semantic segmentation image generated in step S13.

In detail, in step S14 , the particle information acquisition unit 141 may acquire point clusters corresponding to each of a plurality of particles indicating the candidate position of the moving object 1 from the semantic point cloud map 1000 .

Also, in step S14, the projection unit 142 may project the acquired point clusters onto the semantic segmentation image generated by the image processing unit 130, respectively.

In addition, in step S14, the matching score calculation unit 143 calculates the first object classification information assigned to each of the plurality of 3D points included in the point cluster corresponding to each particle and each of the plurality of 3D points of the point cluster. A matching score according to a similarity between second object classification information of the semantic segmentation image assigned to a point projected on the semantic segmentation image may be calculated for each of a plurality of particles.

Also, in step S14, the weight updater 144 may update the weight of each of the plurality of particles based on the calculated matching score.

Also, in step S14, the matching unit 140 may derive position information of the moving object 1 based on the candidate position corresponding to each of the plurality of particles and the weight updated by the weight updating unit 144.

In the foregoing description, steps S11 to S14 may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed.

A location estimation method through holistic matching based on a semantic segmentation image and a semantic point cloud map according to an embodiment of the present application may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. . The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the medium may be those specially designed and configured for the present invention or those known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to act as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the location estimation method through holistic matching based on the semantic segmentation image and the semantic point cloud map described above may be implemented in the form of a computer program or application stored in a recording medium and executed by a computer.

The above description of the present application is for illustrative purposes, and those skilled in the art will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts thereof should be construed as being included in the scope of the present application.

10: mobile body positioning system
100: Location estimation device through holistic matching based on semantic segmentation image and semantic point cloud map
110: map building unit
120: image acquisition unit
130: image processing unit
140: matching unit
141: particle information acquisition unit
142: projection unit
143: matching score calculation unit
144: weight update unit
300: map database
20: network
1: mobile body
11: camera module
1000: semantic point cloud map

Claims (15)

A location estimation method through holistic matching based on a semantic segmentation image and a semantic point cloud map,
Receiving an image input of a target space, which is a space surrounding a predetermined moving object;
generating a semantic segmentation image corresponding to the video input based on a pre-learned first artificial intelligence model to perform object-based semantic segmentation on a predetermined input video; and
Estimating location information of the moving object by matching a pre-built semantic point cloud map with the semantic segmented image;
Including, location estimation method. According to claim 1,
The semantic point cloud map,
A plurality of 3D points representing location information of a preset type of object located in a map construction space including the target space, and constructed in advance such that object classification information for each of the plurality of 3D points is allocated. Phosphorus, location estimation method. According to claim 1,
Estimating the location information,
obtaining a point cluster corresponding to each of a plurality of particles indicating a candidate position of the moving object from the semantic point cloud map;
projecting each of the point clusters onto the semantic segmentation image;
The similarity between the first object classification information allocated to each of the plurality of 3D points included in the point cluster and the second object classification information of the semantic segmentation image allocated to the point where each of the plurality of 3D points is projected calculating a matching score according to the plurality of particles; and
Updating a weight for each of the plurality of particles based on the matching score;
To include, a location estimation method. According to claim 3,
The location information of the mobile body,
Wherein the position estimation method is derived based on the candidate position corresponding to each of the plurality of particles and the updated weight. According to claim 3,
The plurality of particles,
Characterized in that the position according to the change in the state information predicted based on the inertial sensor information of the moving object is a particle predicted to be within a preset distance from the moving object based on satellite navigation information of the moving object. According to claim 5,
The status information is
The location estimation method comprising coordinate information, orientation information, and height information of each of the plurality of particles. According to claim 3,
The step of obtaining the point cluster,
extracting from the semantic point cloud map a plurality of 3D points included in a region of interest set by assuming that the moving object is located at each of the candidate positions;
To include, a location estimation method. According to claim 7,
The step of obtaining the point cluster,
Applying random sampling to a plurality of 3D points included in the region of interest;
To further include, the location estimation method. According to claim 2,
The mobile body is a vehicle,
The preset type of object,
A location estimation method comprising at least one of a lane, a traffic sign, a traffic light, a fence structure, a lighting structure, and a building. According to claim 2,
Prior to receiving the video input,
Building the semantic point cloud map;
Including more,
The building step is
collecting a point cloud including the plurality of 3D points; and
Generating a semantic point cloud corresponding to the collected point cloud based on a pre-learned second artificial intelligence model to perform object-based semantic segmentation on a predetermined input point cloud;
To include, a location estimation method. A location estimation device through holistic matching based on a semantic segmentation image and a semantic point cloud map,
an image acquisition unit receiving an image input of a target space, which is a space surrounding a predetermined moving object;
an image processing unit generating a semantic segmentation image corresponding to the input image based on a first artificial intelligence model learned in advance to perform object-based semantic segmentation on a predetermined input image; and
a matching unit for estimating location information of the moving object by matching a pre-built semantic point cloud map with the semantic segmentation image;
Including, position estimation device. According to claim 11,
The semantic point cloud map,
A plurality of 3D points representing location information of a preset type of object located in a map construction space including the target space, and constructed in advance such that object classification information for each of the plurality of 3D points is allocated. Phosphorus, a localization device. According to claim 11,
The matching unit,
a particle information acquisition unit that obtains a point cluster corresponding to each of a plurality of particles indicating a candidate position of the moving object from the semantic point cloud map;
a projection unit that projects the point clusters onto the semantic segmentation image;
The similarity between the first object classification information allocated to each of the plurality of 3D points included in the point cluster and the second object classification information of the semantic segmentation image allocated to the point where each of the plurality of 3D points is projected a matching score calculator for calculating a matching score according to the plurality of particles; and
a weight update unit for updating a weight for each of the plurality of particles based on the matching score;
To include, the location estimation device. According to claim 13,
The plurality of particles,
Characterized in that the position according to the change of the state information predicted based on the inertial sensor information of the moving object is a particle predicted to be within a preset distance from the moving object based on the satellite navigation information of the moving object. According to claim 11,
Point clouds including the plurality of 3D points are collected, and corresponding to the collected point clouds is based on a pre-learned second artificial intelligence model to perform object-based semantic segmentation on an input predetermined point cloud. A map construction unit for constructing the semantic point cloud map by generating a semantic point cloud to
To further include, the location estimating device.

Images