KR102405767B1 - Apparatus and method for tracking object based on semantic point cloud - Google Patents

Abstract

A semantic point cloud-based object tracking apparatus and method are disclosed, and the semantic point cloud-based object tracking method according to an embodiment of the present application includes: obtaining a semantic point cloud input including a plurality of three-dimensional points; , selecting a target point cloud including a three-dimensional point to which classification information corresponding to a preset object type is assigned from the semantic point cloud input, clustering the target point cloud for each object, and the clustered target point It may include tracking the target object based on the cloud.

Description

Object tracking device and method based on semantic point cloud {APPARATUS AND METHOD FOR TRACKING OBJECT BASED ON SEMANTIC POINT CLOUD}

The present application relates to a semantic point cloud-based object tracking apparatus and method.

The term “smart car” is a fusion of electric, electronic, and information and communication technologies with machine-oriented traditional automobile technology. It has been continuously developed with a focus on aspects, and these smart car-related technologies are specifically designed for safety and It can be classified into related fields and fields related to convenience such as connected cars.

In particular, an intelligent vehicle to which ADAS (Advanced Driver Assistance Systems) or autonomous vehicle is applied is obtained from various sensors (eg, camera, GPS, radar, LiDAR, etc.) provided to recognize the surrounding environment of the driving vehicle. It should be possible to secure the safety of occupants, pedestrians and surrounding vehicles and improve driving convenience by real-time monitoring of the vehicle surroundings with high accuracy through the acquired sensing data.

In particular, LiDAR is a sensor that recognizes the road environment around a car as a three-dimensional shape by measuring the distance to a specific target with a laser to determine the shape of the target object. By providing precise measurement performance, it has the advantage of being applicable to various vehicle-linked recognition systems such as obstacle recognition and dynamic object tracking.

However, with respect to dynamic object tracking and classification using the lidar sensor, the point cloud has a somewhat lower resolution to obtain object classification information compared to a vision sensor such as a camera. It is difficult to select, and due to the characteristics of the lidar sensor, as the distance increases, the density of the point cloud becomes sparse, which reduces the clustering performance and further reduces the accuracy of object tracking. In addition, even when the lidar sensor observes the same object (object), it acquires different shape information depending on the observation point. On the other hand, in the case of dynamic object tracking, since tracking is generally performed based on the center of the object, if the center point is tracked in a different shape, there is a problem in that an incorrect motion state may be tracked.

In addition, the conventional technique of tracking an object around a driving or stopped vehicle using a point cloud obtained through LiDAR, etc. is to obtain classification information that matches each of the three-dimensional points of the point cloud with the type of object associated with the point. Rather than considering only the presence or absence of obstacles around the vehicle by simply clustering 3D points located nearby, or the type of obstacles based on information on the size (eg, cross-sectional area, volume, etc.) of the clustered 3D points, etc. (For example, adjacent vehicles, pedestrians, etc.) were only inferred.

As such, in the conventional object tracking technique that does not consider the classification information of each three-dimensional point of the point cloud, the tracking accuracy is significantly lowered when an association between objects occurs during object tracking, such as when a specific object is located adjacent to another object there was

Accordingly, it goes beyond simply tracking objects around the vehicle by simply using point clouds, which are raw data obtained through LiDAR, etc. A technique that can detect and track is required.

The technology that is the background of the present application is disclosed in Korean Patent Publication No. 10-2103941.

The present application is intended to solve the problems of the prior art described above, and by performing filtering and clustering based on the classification information included in the semantic point cloud input, and then performing object tracking, a predetermined object can be tracked more accurately and quickly. An object of the present invention is to provide a semantic point cloud-based object tracking device and method.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the semantic point cloud-based object tracking method according to an embodiment of the present application includes the steps of obtaining a semantic point cloud input including a plurality of three-dimensional points, the meaning Selecting a target point cloud including a three-dimensional point to which classification information corresponding to a preset object type is assigned from among theoretical point cloud inputs, clustering the target point cloud for each object, and based on the clustered target point cloud to track the target object.

Also, the tracking of the target object may include associating the clustered target point cloud with the target object identified before the semantic point cloud input is obtained.

In addition, the semantic point cloud input may include a plurality of 3D points representing objects existing in an area adjacent to the vehicle.

Also, the object type may include at least one of a vehicle, a pedestrian, and a two-wheeled vehicle.

In addition, the tracking of the target object may include tracking the target object based on a motion model corresponding to the object type, a speed range, and an acceleration range.

In addition, the motion model may include at least one of a constant acceleration model, a constant velocity model, a uniform acceleration model, and a random motion model.

In addition, the selecting of the target point cloud may include dividing and selecting a plurality of target point clouds corresponding to each of a plurality of object types based on the classification information.

In addition, the acquiring of the semantic point cloud input includes: receiving at least one of a point cloud input including a plurality of three-dimensional points for an object in a target space and an image input for the target space; applying semantic region division to at least one of the point cloud input and the image input based on an artificial intelligence model; and generating the semantic point cloud input by reflecting the classification information in the point cloud input.

Meanwhile, the semantic point cloud-based object tracking apparatus according to an embodiment of the present application obtains a semantic point cloud input including a plurality of 3D points, and corresponds to a preset object type among the semantic point cloud inputs A filtering unit that selects a target point cloud including a three-dimensional point to which classification information is assigned, a clustering unit that clusters the target point cloud for each object, and a tracking unit that tracks a target object based on the clustered target point cloud can do.

Also, the tracker may associate the clustered target point cloud with the target object identified before the semantic point cloud input is obtained.

Also, the tracker may track the target object based on a motion model corresponding to the object type, a speed range, and an acceleration range.

Also, the filtering unit may classify and select a plurality of target point clouds respectively corresponding to a plurality of object types based on the classification information.

In addition, the semantic point cloud-based object tracking apparatus according to an embodiment of the present application receives at least one of a point cloud input including a plurality of 3D points for an object in a target space and an image input for the target space and applying semantic region division to at least one of the point cloud input and the image input based on the pre-trained artificial intelligence model to derive the classification information for each of the plurality of 3D points, and the point cloud and a semantic division unit configured to generate the semantic point cloud input by reflecting the classification information in an input.

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, by performing filtering and clustering based on the classification information included in the semantic point cloud input, and then performing object tracking, a semantic object that can be tracked more accurately and quickly A point cloud-based object tracking apparatus and method may be provided.

According to the above-described problem solving means of the present application, even if a predetermined target object is spatially related to another object, the target object and the other object can be independently detected and tracked in consideration of classification information of the object, so that the tracking accuracy can be improved. have.

According to the above-described problem solving means of the present application, object tracking based on a motion model suitable for the type of object to be detected may be performed by using the classification information reflected in the semantic point cloud.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a schematic configuration diagram of an object tracking system including a semantic point cloud-based object tracking apparatus according to an embodiment of the present application.
2 is a diagram exemplarily showing a motion model, a velocity range, and an acceleration range corresponding to an object type.
3 is a conceptual diagram for explaining a process of converting a point cloud input obtained in a target space into a semantic point cloud input.
4 is a schematic configuration diagram of a semantic point cloud-based object tracking apparatus according to an embodiment of the present application.
5 is a flowchart illustrating an object tracking method based on a semantic point cloud according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily carry out. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when a member is positioned “on”, “on”, “on”, “on”, “under”, “under”, or “under” another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

The present application relates to a semantic point cloud-based object tracking apparatus and method.

1 is a schematic configuration diagram of an object tracking system including a semantic point cloud-based object tracking apparatus according to an embodiment of the present application.

1, the object tracking system 10 according to an embodiment of the present application is a semantic point cloud-based object tracking apparatus 100 (hereinafter, 'object tracking apparatus 100') according to an embodiment of the present application. ) and the vehicle 1000 . In addition, although not shown in the drawings, the vehicle 1000 may be provided with a lidar sensor (not shown) that provides the point cloud input 1 to the object tracking apparatus 100 .

The object tracking apparatus 100 and the vehicle 1000 may communicate with each other through the network 20 . The network 20 refers to a connection structure in which information exchange is possible between each node, such as terminals and servers, and an example of such a network 20 includes a 3rd Generation Partnership Project (3GPP) network, a long-term LTE (LTE) network. Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), a wifi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but are not limited thereto.

The object tracking apparatus 100 may obtain a semantic point cloud input 2 including a plurality of three-dimensional points ('Semantic Point Cloud' in FIG. 1). For reference, in the description of the embodiment of the present application, 'point cloud input (1)' includes a plurality of three-dimensional points, but classification information indicating the type of object to which each three-dimensional point corresponds is not processed. It may mean a non-primitive point cloud. In contrast to this, the 'semantic point cloud input (2)' in the present application is a plurality of three-dimensional points included in the raw point cloud input 1, and classification information, which is the type of object corresponding to each of the plurality of three-dimensional points, is a plurality of three-dimensional points. It may mean a point cloud in a processed state to be assigned to each (eg, labeling, etc.).

In other words, classification information indicating the type of object is not assigned to each of the plurality of three-dimensional points included in the point cloud input 1, and each of the plurality of three-dimensional points included in the semantic point cloud input 2 has the above One classification information may be assigned (labeled), and according to the embodiment of the present application, the semantic point cloud input 2 may be generated (acquired) through processing (analysis) of the point cloud input 1 .

Specifically, according to an embodiment of the present application, the object tracking device 100 receives a point cloud input (1) including a plurality of three-dimensional points for an object in a target space, and the received point cloud input (1) By applying semantic segmentation on It may be to obtain a semantic point cloud input (2). On the other hand, as described above, the process of generating the semantic point cloud input 2 by applying the semantic division to the firstly obtained point cloud input 1 will be described in detail later with reference to FIG. 3 .

Meanwhile, in the description of the exemplary embodiment of the present application, the 'target space' may mean an area adjacent to the vehicle 1000 being driven or stopped. In this regard, the object tracking device 100 includes a plurality of points (points) included in the semantic point cloud input 2 reflecting information on at least one object existing in the target space, each object type (class) Classification information indicating , may be obtained from the semantic point cloud 2 .

In addition, with respect to the above-described target space, according to an embodiment of the present disclosure, the object tracking apparatus 100 determines a reference distance set based on the perceived importance in the surrounding area of the vehicle 1000 , and the vehicle 1000 . A region of interest among a plurality of points (points) obtained from a lidar sensor (not shown) using an area within a reference distance from It may be to receive a point cloud input 1 or a semantic point cloud input 2 including a three-dimensional point (point) determined to belong to an area.

In addition, according to an embodiment of the present disclosure, the region of interest corresponds to a driving direction of the vehicle 1000 or a photographing direction of a camera module (not shown) provided in the vehicle 1000 among regions within a reference distance from the vehicle 1000 . area can be set. Here, the photographing direction of the camera module may be referred to as a field of view of the camera module (not shown), a field of view (FoV), or the like.

For example, the reference distance set for setting the region of interest may be 50 m, but is not limited thereto, and the type of the vehicle 1000 , the scale of the road on which the vehicle 1000 is traveling, and the size of the vehicle 1000 . The reference distance may be set differently based on the driving location information and the like, and may be changed while the vehicle 1000 is being driven. Specifically, when the vehicle 1000 is being driven forward, the front region and the lateral region of the vehicle 1000 may be included in the ROI, and the rear region of the vehicle 1000 may not be included in the ROI. Conversely, when the vehicle 1000 is being driven backward, the rear region and the lateral region of the vehicle 1000 may be included in the ROI, and the front region of the vehicle 1000 may not be included as the ROI.

Also, according to an exemplary embodiment of the present disclosure, the reference distance value for setting the region of interest may be set such that the reference distance when the vehicle 1000 moves forward is greater than the reference distance when the vehicle 1000 moves backward. This may be because, when the vehicle 1000 moves forward, the vehicle 1000 typically travels faster than when it moves backward, so object identification over a wider range is required. In this regard, according to an exemplary embodiment of the present disclosure, the reference distance value for setting the ROI may be determined based on driving speed information of the vehicle 1000 . For example, when the driving speed of the vehicle 1000 is fast, the reference distance value for setting the region of interest is set larger than when the driving speed is relatively slow.

Also, referring to FIG. 1 , the object tracking apparatus 100 may perform pre-processing on the received semantic point cloud input 2 ('Pre-processing' in FIG. 1 ). Here, the preprocessing for the semantic point cloud input 2 may include preprocessing for removing some points determined as noise among a plurality of 3D points included in the semantic point cloud input 2 . have.

In addition, the object tracking device 100 is a target point cloud 3 including a 3D point to which classification information corresponding to a preset object type is assigned among a plurality of 3D points included in the semantic point cloud input 2 ) can be selected ('Filtering' in FIG. 1).

According to an embodiment of the present application, each of the plurality of 3D points included in the semantic point cloud input 2 may be displayed with a preset color corresponding to the classification information so that the assigned classification information can be identified. In other words, the semantic point cloud input 2 is Colorized so that a point or space corresponding to a predetermined object type can be visually distinguished in such a way that a color corresponding to the classification information is displayed for each of a plurality of three-dimensional points. ) can be

In addition, according to an embodiment of the present application, classification information that can be reflected in the semantic point cloud input 2 is, for example, a vehicle (Car), a sidewalk (Sidewalk), a road (Road), a building (Building), a two-wheeled vehicle ( Two-wheeled vehicle), bicycle (Bicycle), ground (Ground), nature (Nature), can include several classes (Human), such as. On the other hand, the object tracking apparatus 100 disclosed herein does not perform tracking for all object types that may be included in the semantic point cloud input 2, but may have a key influence on the driving of the vehicle 1000. In order to track only the object corresponding to the object type, the target point cloud 3 including only the 3D point to which the classification information corresponding to the preset object type is allocated may be selectively extracted.

In other words, the object tracking apparatus 100 deletes (excludes) points corresponding to an object classification other than the object classification to be detected, so as to include only points corresponding to the object classification corresponding to the tracking target. can be selected.

Meanwhile, the object type preset for the selection of the target point cloud 3 may include at least one of a vehicle, a pedestrian, and a two-wheeled vehicle. However, the present invention is not limited thereto, and the target point cloud 3 corresponding to additional classification information other than a vehicle, a pedestrian, and a two-wheeled vehicle may be extracted according to an exemplary embodiment of the present disclosure.

More specifically, the object tracking device 100 selects only the three-dimensional point corresponding to the preset object type based on the classification information assigned to each of the plurality of three-dimensional points of the semantic point cloud input 2 and selects the corresponding object It can be created as a target point cloud (3) for a type.

In addition, referring to FIG. 1 , the object tracking apparatus 100 provides a plurality of objects corresponding to each of a plurality of object types based on the classification information allocated to each of the plurality of three-dimensional points of the semantic point cloud input 2 . The point clouds 3a, 3b, and 3c may be separately selected.

In addition, referring to FIG. 1 , the plurality of target point clouds 3a, 3b, and 3c is a first target point cloud ( 3a), a second target point cloud 3b selectively including only a plurality of three-dimensional points to which classification information corresponding to the 'pedestrian' object type is assigned, a plurality of assigned classification information corresponding to the 'two-wheeled vehicle' object type It may include a third target point cloud 3c including only three-dimensional points selectively.

Meanwhile, for each of the plurality of target point clouds 3 that are separately selected for each object type, the process of tracking the target object may be independently performed. In other words, the object tracking apparatus 100 disclosed herein may individually generate the target point cloud 3 corresponding to each object type and perform individual object tracking for each object type.

More specifically, the object tracking apparatus 100 may independently perform a tracking process for a 'vehicle', a tracking process for a 'pedestrian', and a tracking process for a 'two-wheeled vehicle'. In addition, the object tracking device 100 performs a tracking process for each object type, a target point cloud individually generated based on classification information for a corresponding object type among a plurality of 3D points included in the semantic point cloud input 2 . It can be performed based on (3).

Also, the object tracking apparatus 100 may cluster the selected target point cloud 3 for each object ('Clustering' in FIG. 1 ). Here, clustering the target point cloud 3 for each object may mean clustering a plurality of three-dimensional points included in the target point cloud 3 selected in response to a predetermined object type into one object unit. have.

To help understanding, the object tracking device 100 includes a plurality of three-dimensional points corresponding to three different vehicles passing around the vehicle 1000 in the target point cloud 3a corresponding to the 'vehicle' object type. When it is determined to be, the plurality of three-dimensional points of the target point cloud 3a may be clustered into three groups corresponding to each of the three different vehicles. As another example, the object tracking apparatus 100 clusters (groups) three-dimensional points for each pedestrian in response to the 'pedestrian' object type, or clusters the three-dimensional points for each two-wheeled vehicle in response to the 'two-wheeled vehicle' object type (grouping) is possible.

According to an embodiment of the present application, the object tracking apparatus 100 clusters the target point cloud 3 for each object based on a Euclidean clustering technique, a k-means clustering technique, etc. can

Also, according to an embodiment of the present application, the object tracking apparatus 100 may generate a bounding box surrounding each object clustered based on a preset size range for each object type. In other words, the size (eg, area, horizontal length, height, etc.) of the bounding box set corresponding to the 'vehicle' object type and the bounding box set corresponding to the 'pedestrian' object type Box) can be preset differently, and accordingly, the object tracking device 100 uses classification information to set a bounding box that is a spatial range in which object tracking is performed more precisely by using classification information. Objects can be identified.

Also, the object tracking apparatus 100 may track the target object based on the clustered target point cloud 3 . Here, the object tracking device 100 tracking the target object means that the clustered target point cloud is identified before the corresponding semantic point cloud input 2 is obtained (in other words, the currently applied semantic point cloud). It may refer to a process of associating a specific object identified as appearing in the target space for the vehicle 1000) based on the semantic point cloud input (2) applied at a previous point in time in relation to the input (2).

That is, the object tracking apparatus 100 may perform data association based on the target point cloud 3 selected to correspond to each object type. Here, 'data association' can be understood as an operation of classifying whether each currently observed object corresponds to any of the previously tracked (detected) objects, and data predicted through past data and newly observed data It can mean the process of analyzing the relationship between

In this regard, according to the object tracking apparatus 100 disclosed herein, the target object being tracked (eg, vehicle, etc.) is associated with another object (eg, pedestrian, two-wheeled vehicle, fence, etc.) (eg, For example, spatially adjacent), if the object types of the target object and the other object are different, points corresponding to the target object and the other object are separated and selected in different target point clouds 3 , and accordingly, Since the tracking process for the target object and the tracking process for other objects are performed independently based on each target point cloud 3, the target object and the other object are incorrectly detected as one object, or the target object is incorrectly detected as corresponding to the type of another object. False positives that are detected can be prevented.

In other words, since the object tracking device 100 applies the tracking algorithm individually by utilizing the target point cloud 3 that is divided for each object type when performing data association, the possibility of association between objects corresponding to different classification information occurs It can be lowered to improve tracking accuracy.

In addition, according to an embodiment of the present application, the object tracking apparatus 100 includes an Interacting Multiple Model (IMM) filter, an Unscented Kalman Filter (UKF) filter, an Extended Kalman Filter (EKF) filter, and a PDA. A target object may be tracked based on a (Probabilistic Data Association) algorithm, a JPDA (Joint PDA) algorithm, or the like. According to an embodiment of the present application, the object tracking apparatus 100 may perform object tracking for each target point cloud 3 based on an IMM-UKF-PDA based tracker, but is not limited thereto , in the present application, a data association (object tracking) algorithm, a filter, etc. that have been previously known or developed in the future may be variously applied.

Also, the object tracking apparatus 100 may track the target object based on a motion model, a velocity range, and an acceleration range corresponding to each object type of the target point cloud 3 .

2 is a diagram exemplarily showing a motion model, a velocity range, and an acceleration range corresponding to an object type.

Referring to FIG. 2 , the motion model applicable to the 'vehicle' object type or the 'two-wheeled vehicle' object type may include a constant velocity (CV), uniform angular acceleration (CT), and uniform acceleration (CA) motion model. . In addition, the motion model applicable to the 'pedestrian' object type may include a random motion model or the like.

In addition, referring to FIG. 2 , the upper limit speed of the speed range for each object type is the upper limit speed corresponding to the 'two-wheeled vehicle' object type, the upper limit speed corresponding to the 'vehicle' object type, and the upper limit corresponding to the 'pedestrian' object type. It may be determined to be higher in the order of speed. More specifically, the speed range corresponding to the 'vehicle' object type is in the range 0 to 180 km/h, the speed range corresponding to the 'pedestrian' object type is in the range 0 to 45 km/h, and the speed range corresponding to the 'two-wheeled vehicle' object type is The corresponding speed range may be determined in the range of 0 to 300 km/h, but is not limited thereto.

Also, referring to FIG. 2 , the upper limit value of the acceleration range for each object type may be determined such that the upper limit speed corresponding to the 'two-wheeled vehicle' object type is higher than the upper limit value corresponding to the 'vehicle' object type. More specifically, the acceleration range corresponding to the 'vehicle' object type may be determined to be in the range of 0 to 24 km/s ² , and the speed range corresponding to the 'two-wheeled vehicle' object type may be determined to be in the range of 0 to 25 km/s ² . but is not limited.

As such, the object tracking device 100 disclosed herein compensates for the absence of classification information for each object type in the point cloud input 1 in the raw state by utilizing the semantic point cloud input 2, thereby providing an object (object). When tracking, by customizing the motion model and movement pattern (speed, acceleration, etc.) according to the classification of each object (object), faster and more accurate object tracking can be achieved.

In addition, according to an embodiment of the present application, the object tracking apparatus 100 sets the model transition probability of the IMM-UKF-PDA algorithm for object tracking based on the classification information (object type) of each target point cloud 3 . can

As such, the conventional object tracking algorithm merely performs object tracking based on the same parameters without considering the movement characteristics of each object type that may be located around the vehicle 1000, such as a vehicle, a pedestrian, a two-wheeled vehicle, etc. The object tracking apparatus 100 disclosed in , may perform object tracking based on various parameters that are customized to suit an object type.

Hereinafter, with reference to FIG. 3 , the object tracking device 100 acquires classification information based on the point cloud input 1 in the raw state to which the classification information is not assigned, and reflects the semantic point cloud input 2 Let us explain the process of creating (processing).

3 is a conceptual diagram for explaining a process of converting a point cloud input obtained in a target space into a semantic point cloud input.

Referring to FIG. 3 , the object tracking apparatus 100 may receive at least one of a point cloud input 1 including a plurality of 3D points for an object in a target space and an image input 1 ′ for the target space. can For example, the object tracking device 100 may receive only the three-dimensional point cloud input 1 or receive the point cloud input 1 and the image input 1 ′ corresponding to the point cloud input 1 together. have.

In addition, the object tracking apparatus 100 may apply semantic region division to at least one of the point cloud input 1 and the image input 1 ′ based on the pre-trained artificial intelligence model. That is, the object tracking apparatus 100 derives classification information for each of a plurality of three-dimensional points included in the point cloud input 1 based on the semantic region division result, and applies the derived classification information to the point cloud input 1 ) to generate a semantic point cloud input (2).

According to an embodiment of the present application, the object tracking apparatus 100 generates a projection image corresponding to the point cloud input 1 by projecting the received three-dimensional point cloud input 1 in two dimensions, and the converted projection A semantic point cloud input (2) is generated by applying artificial intelligence-based semantic segmentation to an image to generate a two-dimensional semantic segmentation image, and reconstructing the generated two-dimensional semantic segmentation image in three dimensions. can operate to do so.

Here, the AI-based model for generating a two-dimensional semantic segmentation image based on the projection image is, for example, a plurality of encoder-side layers that perform downsampling on the input projection image. and a plurality of decoder-side layers outputting a two-dimensional semantic segmentation image corresponding to the input predetermined image by performing upsampling on the down-sampled predetermined image. can

Specifically, the artificial intelligence model possessed by the object tracking apparatus 100 may be designed in a structure that does not include a skip connection between an encoder-side layer and a decoder-side layer. Specifically, skip connection can be understood as a technique for adding features from downsampling in upsampling, and in the layer of the latter part of the AI model, not only the output from the previous layer of the corresponding layer but also the first part of the AI model. It may mean skipping the middle layer and connecting the layer of the first half and the layer of the second half so that the signal from the arranged layer can be utilized.

In the case of an AI model with a structure that includes a skip connection (for example, a model using a Darknet-based backbone room, etc.), skip connection can prevent signal attenuation during the backpropagation operation. Therefore, even if the layers of the AI model become deeper (in other words, even if the number of layers increases), there may be an effect of efficiently performing learning, but the use of memory to transmit signals from the first layer to the second layer It is time consuming and can slow down the reasoning speed of the AI model.

Accordingly, the object tracking apparatus 100 according to an embodiment of the present application may train an artificial intelligence model having a structure that does not include a skip connection so as to improve the inference speed of the semantic segmentation process.

In addition, the number (m) of the decoder-side layers of the artificial intelligence model possessed by the object tracking apparatus 100 according to an embodiment of the present application may be smaller than the number (n) of the encoder-side layers (m<n).

In this regard, the decoder-side layer serves to restore the characteristics of the input image compressed by the encoder through up-sampling, and the operation of the decoder-side layer is linked with the fine-tuning level of the parameters of the artificial intelligence model. Therefore, if the performance above a predetermined level is satisfied, there is no need to deeply stack the decoder-side layer in order to quickly generate an output image through the entire architecture, so the object tracking apparatus 100 according to an embodiment of the present application is semantic segmentation In order to improve the inference speed of the process, the AI model can be trained with a structure in which the number of layers on the decoder side is smaller than the number of layers on the encoder side.

In addition, according to another embodiment of the present application, the object tracking apparatus 100 utilizes both the point cloud input 1 and the image input 1', and based on the semantic region segmentation result for the image input 1' Accordingly, classification information may be derived in a manner of classifying each point cluster obtained by grouping a plurality of 3D points included in the point cloud input based on the object.

In other words, the object tracking apparatus 100 according to another embodiment of the present application is a cluster data obtained by clustering a plurality of 3D points included in the point cloud input 1 according to a predetermined criterion and dips the image input 1 ′. Object classification data for each pixel of the image input 1' is derived by applying learning-based semantic region division, respectively, and an object for each point cluster of cluster data based on the derived cluster data and object classification data Classification information can be derived by selecting a label.

More specifically, the object tracking apparatus 100 according to another embodiment of the present application synchronizes the cluster data derived from the point cloud input 1 and the object classification data derived from the image input 1 ′, and preset calibration parameters After mapping the synchronized cluster data to a predetermined image coordinate system using Classification information can be derived.

In summary, the object tracking apparatus 100 according to another embodiment of the present application is a real-time fusion of an image captured by a camera and a three-dimensional point cloud obtained by LiDAR through an artificial function-based image processing technique. It is possible to generate a semantic point cloud input (2).

4 is a schematic configuration diagram of a semantic point cloud-based object tracking apparatus according to an embodiment of the present application.

Referring to FIG. 4 , the object tracking apparatus 100 may include a semantic division unit 110 , a filtering unit 120 , a clustering unit 130 , and a tracking unit 140 .

The semantic division unit 110 may receive at least one of a point cloud input 1 including a plurality of 3D points for an object in the target space and an image input 1 ′ for the target space. In addition, the semantic division unit 110 may apply semantic domain division to at least one of the point cloud input 1 and the image input 1 ′ based on a pre-trained artificial intelligence model. Also, the semantic division unit 110 may derive classification information for each of the plurality of 3D points based on the semantic domain division result. Also, the semantic division unit 110 may generate the semantic point cloud input 2 by reflecting the classification information derived from the point cloud input 1 .

The filtering unit 120 obtains a semantic point cloud input 2 including a plurality of 3D points, and a preset object type among a plurality of 3D points included in the obtained semantic point cloud input 2 . The target point cloud 3 including the 3D point to which the classification information corresponding to is assigned may be selected.

The clustering unit 130 may cluster the selected target point cloud 3 for each object.

The tracker 140 may track the target object based on the clustered target point cloud 3 .

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

5 is a flowchart illustrating an object tracking method based on a semantic point cloud according to an embodiment of the present application.

The method for tracking an object based on a semantic point cloud according to an embodiment of the present application shown in FIG. 5 may be performed by the object tracking apparatus 100 described above. Accordingly, even if omitted below, the description of the object tracking apparatus 100 may be equally applied to the description of the semantic point cloud-based object tracking method.

Referring to FIG. 5 , in step S11 , the filtering unit 120 may obtain a semantic cloud input 2 . According to an embodiment of the present application, in step S11 , the filtering unit 120 may receive the semantic point cloud input 2 generated by the semantic division unit 110 from the semantic division unit 110 . .

More specifically, before step S11, the semantic division unit 110 performs at least one of a point cloud input 1 including a plurality of three-dimensional points for an object in the target space and an image input 1' for the target space. can receive

In addition, before step S11, the semantic segmentation unit 110 applies semantic region segmentation to at least one of the received point cloud input 1 and the image input 1' based on the pre-trained artificial intelligence model. can

Also, before step S11 , the semantic division unit 110 may derive classification information for each of the plurality of 3D points included in the point cloud input 1 based on the result of the semantic domain division.

Also, before step S11 , the semantic division unit 110 may generate the semantic point cloud input 2 by reflecting the classification information derived in the received point cloud input 1 .

Next, in step S12, the filtering unit 120 selects a target point cloud 3 including a three-dimensional point to which classification information corresponding to a preset object type is assigned from among the acquired semantic point cloud inputs 2 . can

According to an embodiment of the present application, in step S12 , the filtering unit 120 performs a plurality of object types corresponding to each of the plurality of object types based on the classification information of each of the plurality of 3D points included in the semantic point cloud input 2 . Each of the target point clouds 3 may be separately selected.

Next, in step S13 , the clustering unit 130 may cluster the selected target point cloud 3 for each object.

Next, in step S14 , the tracking unit 140 may track the target object based on the clustered target point cloud.

Specifically, in step S14 , the tracker 140 may associate at least one object included in the clustered target point cloud with the target object identified before the semantic point cloud input 2 is obtained.

In addition, according to an embodiment of the present application, in step S14 , the tracker 140 performs the target object based on the motion model, velocity range, and acceleration range corresponding to the object type of at least one object included in the clustered target point cloud. can be tracked.

In the above description, steps S11 to S14 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between the steps may be changed.

The semantic point cloud-based object tracking method according to an embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the above-described semantic point cloud-based object tracking method may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The above description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

10: Object tracking system
100: semantic point cloud-based object tracking device
110: semantic division
120: filtering unit
130: clustering unit
140: tracking unit
1000: vehicle
1: Point cloud input
2: Semantic point cloud input
20: network

Claims (14)

In the semantic point cloud-based object tracking method performed by the semantic point cloud-based object tracking device,
obtaining a semantic point cloud input comprising a plurality of three-dimensional points;
selecting a target point cloud including a 3D point to which classification information corresponding to a preset object type is assigned from among the semantic point cloud inputs;
clustering the target point cloud for each object; and
tracking a target object based on the clustered target point cloud;
including,
The step of tracking the target object,
and tracking the target object by associating the clustered target point cloud with the target object identified as appearing in the target space before the semantic point cloud input is obtained. delete According to claim 1,
The semantic point cloud input includes a plurality of three-dimensional points representing objects existing in an area adjacent to the vehicle,
Wherein the object type includes at least one of a vehicle, a pedestrian, and a two-wheeled vehicle. 4. The method of claim 3,
The step of tracking the target object,
and tracking the target object based on a motion model, a velocity range, and an acceleration range corresponding to the object type. 5. The method of claim 4,
The motion model is
An object tracking method comprising at least one of a constant acceleration model, a constant velocity model, a uniform acceleration model, and a random motion model. 4. The method of claim 3,
The step of selecting the target point cloud,
The object tracking method, wherein the plurality of target point clouds corresponding to each of the plurality of object types are separately selected based on the classification information. According to claim 1,
The step of obtaining the semantic point cloud input comprises:
receiving at least one of a point cloud input including a plurality of 3D points for an object in the target space and an image input for the target space;
applying semantic region division to at least one of the point cloud input and the image input based on a pre-trained artificial intelligence model;
deriving the classification information for each of the plurality of 3D points based on the semantic region division result; and
generating the semantic point cloud input by reflecting the classification information in the point cloud input;
A method for tracking an object, comprising: In a semantic point cloud-based object tracking device,
A filtering unit that obtains a semantic point cloud input including a plurality of three-dimensional points, and selects a target point cloud including a three-dimensional point to which classification information corresponding to a preset object type is assigned from among the semantic point cloud inputs ;
a clustering unit clustering the target point cloud for each object; and
a tracking unit for tracking a target object based on the clustered target point cloud;
including,
The tracking unit,
The object tracking apparatus of claim 1, wherein the target object is tracked by associating the clustered target point cloud with the target object identified as appearing in the target space before the semantic point cloud input is obtained. delete 9. The method of claim 8,
The semantic point cloud input includes a plurality of three-dimensional points representing objects existing in an area adjacent to the vehicle,
The object type will include at least one of a vehicle, a pedestrian, and a two-wheeled vehicle, the object tracking device. 11. The method of claim 10,
The tracking unit,
tracking the target object based on a motion model, a velocity range, and an acceleration range corresponding to the object type;
The motion model is
An object tracking device comprising at least one of a constant acceleration model, a constant velocity model, a uniform acceleration model, and a random motion model. 11. The method of claim 10,
The filtering unit,
Based on the classification information, the object tracking apparatus of the plurality of target point clouds corresponding to each of the plurality of object types to be separately selected. 9. The method of claim 8,
Receive at least one of a point cloud input including a plurality of 3D points for an object in the target space and an image input for the target space, and the point cloud input and the image input based on a pre-trained artificial intelligence model applying semantic domain division to at least one of the plurality of 3D points to derive the classification information, and reflecting the classification information to the point cloud input to generate the semantic point cloud input division,
Which will further include an object tracking device. A computer-readable recording medium recorded as a program for executing the method according to any one of claims 1 to 7 on a computer.

Images