KR102451797B1 - Modeling method, modeling device and modeling system for modeling target object - Google Patents

Abstract

The present invention relates to a modeling method, a modeling apparatus, and a modeling system for modeling a target object. The modeling method, according to the present invention, comprises the steps of: obtaining first target lidar points; obtaining second target lidar points; generating a plurality of planes; converting coordinates of the first target lidar points; and modeling a target object. Accordingly, an object can be modeled even when the point density of a lidar point cloud is sparse.

Description

A modeling method for modeling a target object, a modeling device, and a modeling system

The present application relates to a modeling method, a modeling apparatus, and a modeling system for modeling a target object, and more particularly, to a modeling method, a modeling apparatus, and a modeling system for modeling a target object by accumulating lidar point clouds.

Recently, the development of technology for modeling an object is actively being made. As sensors for object modeling, cameras, lidars, radars, etc. are used.

Dual lidar is widely used due to its advantages such as spatial resolution and excellent night performance. Accordingly, various technologies for modeling objects using lidar point clouds are being developed.

However, there is a problem in that it is difficult to model an object when the lidar point is sparse or extremely sparse by using the conventional lidar point registration method such as ICP or G-ICP.

One problem to be solved in the present application is to provide a modeling method, a modeling apparatus, and a modeling system capable of modeling an object even when the point density of the lidar point cloud is sparse.

An object to be solved in the present application is to provide a modeling method, a modeling apparatus, and a modeling system capable of modeling the object by reflecting the movement of the object.

The problems to be solved in the present application are not limited to the above-described problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present application belongs from the present application.

According to an embodiment, in the modeling method of modeling a target object performed by a computing device, the steps of obtaining a first image frame and a first lidar point cloud, the first image frame and the first lidar point the cloud corresponds to the first point in time; obtaining first target characteristic information related to the target object from the first image frame, wherein the first target characteristic information reflects a shape of the target object at the first viewpoint; determining one or more first target lidar points associated with the target object from among the first lidar point clouds; acquiring a second image frame and a second lidar point cloud, wherein the second image frame and the second lidar point cloud correspond to a second time point that is after the first time point; obtaining second target characteristic information related to the target object from the second image frame, wherein the second target characteristic information reflects a shape of the target object at the second viewpoint; determining one or more second target lidar points associated with the target object from among the second lidar point clouds; Reflecting the movement of the target object between the first viewpoint and the second viewpoint based on the first target characteristic information acquired from the first image frame and the second target characteristic information acquired from the second image frame obtaining conversion information; transforming the coordinates of the first target lidar points based at least in part on the transformation information obtained from the first image frame and the second image frame; and modeling the target object using the converted first target LIDAR points and the second target LIDAR points.

According to an embodiment, in a modeling apparatus for modeling a target object, the communication unit for receiving an image frame and lidar point cloud; and a control unit for modeling the target object using the image frame and lidar point cloud received through the communication unit, wherein the control unit acquires a first image frame and a first lidar point cloud through the communication module and - the first image frame and the first lidar point cloud correspond to a first viewpoint - obtain first target characteristic information related to the target object from the first image frame, - the first target characteristic information reflecting the shape of the target object at the first time point - determining one or more first target LIDAR points related to the target object among the first LIDAR point clouds, and a second image frame through the communication module and obtaining a second lidar point cloud, wherein the second image frame and the second lidar point cloud correspond to a second viewpoint that is after the first viewpoint, wherein the second image frame is associated with the target object. obtain second target characteristic information, wherein the second target characteristic information reflects a shape of the target object at the second viewpoint; one or more second targets related to the target object in the second lidar point cloud LiDAR points are determined, and based on the first target characteristic information obtained from the first image frame and the second target characteristic information obtained from the second image frame, between the first time point and the second time point. Obtaining transformation information reflecting the movement of the target object, transforming the coordinates of the first target LiDAR points based at least in part on the transformation information obtained from the first image frame and the second image frame, and A modeling apparatus for modeling the target object using the converted first target LIDAR points and the second target LIDAR points may be provided.

According to one embodiment, a modeling system for modeling a target object, comprising: a camera module that generates a first image frame corresponding to a first viewpoint and a second image frame corresponding to a second viewpoint that is after the first viewpoint; a lidar module that generates a first lidar point cloud corresponding to the first viewpoint and a second lidar point cloud corresponding to the second viewpoint; and a modeling apparatus for modeling the target object using the image frames and the lidar point clouds, wherein the modeling apparatus obtains the first and second image frames from the camera module, and obtain first target characteristic information related to the target object from an image frame, wherein the first target characteristic information reflects a shape of the target object at the first viewpoint; and the target object from the second image frame obtain related second target characteristic information, wherein the second target characteristic information reflects the shape of the target object at the second time point, and obtain the first and second lidar point clouds from the lidar module and determining one or more first target lidar points related to the target object from among the first lidar point clouds, and determining one or more second target lidar points related to the target object from among the second lidar point clouds. and, based on the first target characteristic information acquired from the first image frame and the second target characteristic information acquired from the second image frame, the movement of the target object between the first viewpoint and the second viewpoint obtain transformation information reflecting A modeling system for modeling the target object using lidar points and the second target lidar points may be provided.

According to an embodiment, there is provided a modeling method for modeling a target object performed by one or more processors, comprising: acquiring first target lidar points corresponding to a first viewpoint; obtaining second target lidar points corresponding to a second viewpoint different from the first viewpoint; For each of the second target lidar points, generating a plurality of planes associated with each second target lidar point by using the respective second target lidar point and the second target lidar points adjacent thereto ; The first target lidar points, the second target lidar points, and transform the coordinates of the first target lidar points using transformation information calculated based on the plurality of planes, the transformation information reflecting movement between the first and second viewpoints of the target object; and modeling the target object using the transformed first target lidar points and the second target lidar points, wherein transforming the coordinates includes, for each of the first target lidar points, determining a plane of the plurality of planes corresponding to each of the first target lidar points; calculating the transformation information based on the first target LIDAR points and corresponding planes; and applying the transformation information to the coordinates of the first target lidar points to repeatedly transform the coordinates of the first target lidar points until a predetermined condition is satisfied. .

The solutions of the problems of the present application are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present application belongs from the present application.

According to the embodiment of the present application, even when the point density of the lidar point cloud is sparse, it is possible to provide a modeling method, a modeling apparatus, and a modeling system capable of modeling an object by accumulating the point density.

According to an embodiment of the present application, a modeling method, a modeling apparatus, and a modeling system for modeling an object through coordinate transformation of a lidar point cloud using transformation information obtained from an image may be provided.

Effects of the invention of the present application are not limited to the above-described effects, and effects not mentioned will be clearly understood by those of ordinary skill in the art to which this application belongs from this application.

1 is a diagram illustrating target object modeling through accumulation of target lidar points according to an embodiment.
2 is a diagram of a moving target object modeling through accumulation of target lidar points according to an embodiment.
3 is a diagram of a modeling system according to an embodiment.
4 is a diagram of a modeling apparatus according to an exemplary embodiment.
5 is a diagram illustrating a modeling method using transformation information calculated based on an image frame according to an exemplary embodiment.
6 is a diagram of image coordinates according to an exemplary embodiment.
7 is a diagram of a target lidar point according to an embodiment.
8 is a diagram for explaining conversion information according to an embodiment.
9 is a diagram for obtaining conversion information according to an embodiment.
10 is a diagram of examples of a transform information calculation algorithm.
11 is a diagram for converting first coordinates of a first target lidar point into second coordinates according to an embodiment.
12 and 13 are diagrams for explaining a time period in which target LiDAR points are accumulated, according to an embodiment.
14 is a diagram for describing object type information generated based on an image frame according to an exemplary embodiment.
15 is a diagram for determining a target lidar point from among lidar point clouds by using object type information according to an embodiment.
16 is a diagram illustrating obtaining target characteristic information from characteristic information using object type information according to an embodiment.
17 is a diagram related to voxelization according to an embodiment.
18 is a diagram illustrating a modeling method using transformation information calculated based on a lidar point according to an embodiment.
19 is a diagram for generating planes using lidar points according to an embodiment.
20 is a diagram illustrating coordinate transformation of lidar points according to an embodiment.
21 is a diagram illustrating an example of an algorithm for a modeling method using transformation information calculated based on a lidar point.
22 is a table regarding the accuracy of SP-ICP, which is an embodiment of a modeling method.
23 and 24 are diagrams illustrating visualization of a modeling result using SP-ICP, which is an embodiment of a modeling method.
25 is a more numerical representation of the accuracy of the test results of FIGS. 23 and 24 .

The embodiments described in the present application are for clearly explaining the idea of the present application to those of ordinary skill in the technical field to which the present application belongs, so it is not limited by the embodiments described in the present application, and the scope of the present application is It should be construed as including modifications or variations without departing from the spirit of the present application.

The terms used in the present application have been selected as widely used general terms as possible in consideration of the functions in the present application, but they may vary depending on the intention, custom, or emergence of new technology of those of ordinary skill in the art to which this application belongs. can However, when a specific term is defined and used in an arbitrary sense, the meaning of the term will be separately described. Therefore, the terms used in the present application should be interpreted based on the actual meaning of the term and the overall content of the present application, rather than the simple name of the term.

The drawings of the present application are for easy explanation of the present application, and the shapes shown in the drawings may be exaggerated as necessary to help understand the present application, so the present application is not limited by the drawings.

In the present application, when it is determined that a detailed description of a known configuration or function related to the present application may obscure the gist of the present application, a detailed description thereof will be omitted if necessary. In addition, the numbers (eg, first, second, etc.) used in the description of the present application are merely identification symbols for distinguishing one component from other components.

According to an embodiment, in the modeling method of modeling a target object performed by a computing device, the modeling method includes: obtaining a first image frame and a first lidar point cloud; the first lidar point cloud corresponds to the first time point; obtaining first target characteristic information related to the target object from the first image frame, wherein the first target characteristic information reflects a shape of the target object at the first viewpoint; determining one or more first target lidar points associated with the target object from among the first lidar point clouds; acquiring a second image frame and a second lidar point cloud, wherein the second image frame and the second lidar point cloud correspond to a second time point that is after the first time point; obtaining second target characteristic information related to the target object from the second image frame, wherein the second target characteristic information reflects a shape of the target object at the second viewpoint; determining one or more second target lidar points associated with the target object from among the second lidar point clouds; Reflecting the movement of the target object between the first viewpoint and the second viewpoint based on the first target characteristic information acquired from the first image frame and the second target characteristic information acquired from the second image frame obtaining conversion information; transforming the coordinates of the first target lidar points based at least in part on the transformation information obtained from the first image frame and the second image frame; and modeling the target object using the converted first target LIDAR points and the second target LIDAR points.

The transformation information may be information for transforming two-dimensional coordinates, and the coordinates of the first target lidar points that are transformed based on the transformation information may be three-dimensional coordinates.

The modeling method includes: obtaining calibration information between a camera module generating the image frames and a lidar module generating the lidar point clouds - The calibration information is a relative position between the camera module and the lidar module At least one of translation information reflecting the, rotation information reflecting the relative direction between the camera module and the lidar module, and camera calibration information for projection onto the image plane of the camera module It may further include, and in transforming the coordinates of the first target lidar points, the coordinates of the first target lidar points may be transformed further based on the calibration information.

The converting of the coordinates of the first target LiDAR points may include projecting the first target LiDAR points having first coordinates corresponding to the first viewpoint on the image plane based on the calibration information on the image plane. obtaining image coordinates corresponding to one coordinate; converting the coordinate values of the image coordinates by applying the transformation information to the image coordinates; and generating second coordinates corresponding to the second viewpoints of the first target lidar points from the converted image coordinates based on the calibration information.

The second coordinates may be determined as specific coordinates having the same height value as the first coordinates among a group of coordinates mapped with the transformed image coordinates.

The converting of the coordinates of the first target LIDAR points may include converting the coordinates of the first target LIDAR points assuming that the target object moves on a predetermined plane.

The obtaining of the transformation information may include: matching the first target characteristic information and the second target characteristic information; calculating the transformation information based on the matched first target characteristic information and second target characteristic information; converting image coordinates of the first target characteristic information by applying the transformation information to the first target characteristic information; and comparing the converted first target characteristic information and the second target characteristic information.

The modeling method may include: generating first object type information reflecting a first target area related to the target object in the first image frame; and generating second object type information reflecting a second target area related to the target object in the second image frame.

The obtaining of the first target characteristic information may include: generating first characteristic information from the first image frame; and determining, among the generated first characteristic information, characteristic information corresponding to the first target area as the first target characteristic information, by using the first object type information, and the second target The obtaining of the characteristic information may include: generating second characteristic information from the second image frame; and determining, as the second target characteristic information, characteristic information corresponding to the second target area from among the generated second characteristic information, by using the second object type information.

The determining of the first target LIDAR points may include selecting one or more LIDAR points corresponding to the first target area from among the first LIDAR point clouds as the first target LIDAR by using the first object type information. The method may include determining the IDA points, wherein the determining of the second target LIDAR points may include: using the second object type information to be located in the second target area of the second LIDAR point cloud. and determining corresponding one or more lidar points as the second target lidar points.

The first time point and the second time point may be included in a predetermined time interval.

The transformation information may include a homography matrix generated based on the first target characteristic information and the second target characteristic information.

According to one embodiment, in the modeling apparatus for modeling a target object, the modeling apparatus, the communication unit for receiving the image frame and the lidar point cloud; and a control unit for modeling the target object using the image frame and lidar point cloud received through the communication unit, wherein the control unit includes a first image frame and a first lidar point cloud through the communication module obtain - the first image frame and the first lidar point cloud correspond to a first viewpoint - obtain first target characteristic information related to the target object from the first image frame - the first target characteristic information reflects the shape of the target object at the first point in time; determines one or more first target LIDAR points associated with the target object among the first LIDAR point clouds; obtain an image frame and a second lidar point cloud, wherein the second image frame and the second lidar point cloud correspond to a second viewpoint that is after the first viewpoint, the target object from the second image frame obtain second target characteristic information related to - the second target characteristic information reflects the shape of the target object at the second time point; 2 target LiDAR points are determined, and the first viewpoint and the second viewpoint based on the first target characteristic information acquired from the first image frame and the second target characteristic information acquired from the second image frame obtaining transformation information reflecting the movement of the target object between , the target object may be modeled using the converted first target LIDAR points and the second target LIDAR points.

According to an embodiment, in the modeling system for modeling a target object, the modeling system is configured to generate a first image frame corresponding to a first viewpoint and a second image frame corresponding to a second viewpoint that is after the first viewpoint. generating camera module; a lidar module that generates a first lidar point cloud corresponding to the first viewpoint and a second lidar point cloud corresponding to the second viewpoint; and a modeling apparatus for modeling the target object using the image frames and the lidar point clouds, wherein the modeling apparatus obtains the first and second image frames from the camera module, obtain first target characteristic information related to the target object from a first image frame, wherein the first target characteristic information reflects a shape of the target object at the first viewpoint; obtain second target characteristic information related to an object, wherein the second target characteristic information reflects the shape of the target object at the second time point, and the first and second lidar point clouds from the lidar module , determine one or more first target lidar points associated with the target object in the first lidar point cloud, and one or more second target lidar points associated with the target object in the second lidar point cloud. determine the target object between the first viewpoint and the second viewpoint based on the first target characteristic information acquired from the first image frame and the second target characteristic information acquired from the second image frame obtain transformation information reflecting the movement of The target object may be modeled using the first target lidar points and the second target lidar points.

According to an embodiment, in a modeling method for modeling a target object performed by one or more processors, the modeling method includes: acquiring first target LIDAR points corresponding to a first viewpoint; obtaining second target lidar points corresponding to a second viewpoint different from the first viewpoint; For each of the second target lidar points, generating a plurality of planes associated with each second target lidar point by using the respective second target lidar point and the second target lidar points adjacent thereto ; The first target lidar points, the second target lidar points, and transform the coordinates of the first target lidar points using transformation information calculated based on the plurality of planes, the transformation information reflecting movement between the first and second viewpoints of the target object; and modeling the target object using the transformed first target lidar points and the second target lidar points, wherein transforming the coordinates includes each of the first target lidar points every time, determining a plane corresponding to the respective first target lidar point of the plurality of planes; calculating the transformation information based on the first target LIDAR points and corresponding planes; and applying the transformation information to the coordinates of the first target lidar points to repeatedly transform the coordinates of the first target lidar points until a predetermined condition is satisfied.

The second target LIDAR points may include a first target point and a second target point, and the plurality of planes are generated using the first target point and the second target LIDAR points adjacent thereto and the It may include a plurality of first planes related to the first target point and a plurality of second planes generated using the second target point and the second target lidar points adjacent thereto and related to the second target point. .

The generating of the plurality of planes may include merging at least some of the plurality of first planes into one plane in consideration of a degree of similarity between the plurality of first planes.

The determining of the plane may include: for each of the first target LiDAR points, determining a point corresponding to each of the first target LiDAR points among the second target LiDAR points; and determining a plane corresponding to each of the first target LiDAR points from among one or more planes related to each of the first target LiDAR points and corresponding points.

The present application discloses a modeling method for modeling an object, a modeling apparatus, and a modeling system. Although the present application discloses a modeling method, a modeling apparatus, and a modeling system for modeling an object by accumulating lidar points whose positions are expressed in three-dimensional coordinates, the present application is not limited thereto, and the present application is not limited thereto. It can also be applied to modeling an object by accumulating data expressed in coordinates.

In addition, although the present application mainly describes maritime conditions such as operation or anchoring of a ship, the present application is not limited thereto, and may be similarly applied to other cases such as road conditions such as driving of a vehicle.

In the present application, an object to be modeled is referred to as a target object. In addition, in the present application, a lidar point corresponding to the target object is referred to as a target lidar point. In the present application, that the lidar point corresponds to a specific object may mean that the lidar point is reflected from the specific object or is predicted to be reflected.

In the present application, that a lidar point (or lidar point cloud, target lidar point) corresponds to a specific point in time or a specific time interval means that the lidar point (or lidar point cloud, target lidar point) corresponds to the lidar module by the lidar module. ) may mean generated at the specific time point or within the specific time interval. Alternatively, that the lidar point (or lidar point cloud, target lidar point) corresponds to a specific time point or a specific time period means that the modeling device selects the lidar point (or lidar point cloud, target lidar point) as the specific It may mean acquired at a time point or within the specific time period.

In the present application, that the image frame corresponds to a specific time point or a specific time period may mean that the image frame is generated at the specific time point or within the specific time period by the camera module. Alternatively, that the image frame corresponds to a specific time point or a specific time period may mean that the modeling apparatus acquires the image frame at the specific time point or within the specific time period.

According to embodiments of the present application, a modeling method, a modeling apparatus, and a modeling system for modeling a target object by accumulating target LiDAR points are disclosed.

1 is a diagram of a target object modeling through accumulation of target LiDAR points according to an embodiment, and relates to modeling a target object by accumulating target LiDAR points corresponding to a non-moving target object. In this case, that the target object does not move may mean that there is no relative movement with respect to the lidar module.

Referring to FIG. 1 , the lidar point clouds obtained at each of the first time point t ₁ , the second time point t ₂ , and the third time point t ₃ are target LIDAR points corresponding to the target object 10 . It may include black filled circles (1a, 1b, 1c) and LiDAR points (empty circles) that do not correspond to the target object 10 . In this case, the target object 10 may be modeled by accumulating the target lidar points 1a, 1b, and 1c over time. More specifically, target LIDAR points 1a are selected from the LIDAR point clouds obtained at the first time point t ₁ , and target LIDAR points 1a are selected from the LIDAR point clouds obtained at the second time point t ₂ . Accumulating the IDA points 1b at the target LIDAR points 1a selected at the first time point t ₁ to obtain the accumulated target LIDAR points 1d at the second time point t ₂ , , by accumulating target lidar points 1c from the lidar point cloud acquired at the third time t ₃ to the accumulated target lidar points 1d acquired at the second time t ₂ , The accumulated target LIDAR points 1e of the third time point t ₃ may be obtained. At this time, the target LiDAR points 1a at the first time point t ₁ , the accumulated target LiDAR points 1d at the second time point t ₂ , and the accumulated target LIDAR points 1d at the third time point t ₃ ) The target lidar points 1e each model the target object 10 . However, as the time for accumulating the target LIDAR points increases, the target object 10 may be modeled in more detail. That is, the accumulated target LiDAR points 1d at the second time point t ₂ provide more detailed information about the target object 10 than the target LiDAR points 1a at the first time point t ₁ . modeled, and the accumulated target LiDAR points 1e at the third time point t ₃ are compared to the accumulated target LiDAR points 1d at the second time point t ₂ , the target object 10 can be modeled in more detail.

Meanwhile, in order to accumulate target lidar points corresponding to the moving target object, information on the movement of the target object over time is required. In other words, in order to model a moving target object by accumulating target LiDAR points, information on the movement of the target object over time is required. In this case, the movement of the target object may mean that there is a relative movement with respect to the lidar module (eg, a change in the position of the target object, a change in the orientation of the target object, etc.). In this case, information on the movement of the target object should be reflected in target lidar points and the target lidar points should be accumulated. Compared with the case in which the target object does not move, it is necessary to not only select a target lidar point corresponding to the target object, but also determine the movement of the target lidar point corresponding to the target object over time. For example, in order to acquire target LIDAR points accumulated up to a specific point in time, locations of target LIDAR points of a time prior to the specific point in time should be identified at the specific point in time. That is, it is necessary to know the coordinates of the target lidar points of the previous time point at the specific time point. To this end, the coordinates of the target lidar points of the previous time should be converted into coordinates of the specific time.

2 is a diagram of a moving target object modeling through accumulation of target lidar points according to an embodiment.

Referring to (a) of FIG. 2 , the target object 20 moves as it passes through the first time point t ₁ , the second time point t ₂ , and the third time point t ₃ , and the The lidar point clouds include target lidar points (black filled circles, 2a, 2b, 2c) corresponding to the target object 20 and lidar points not corresponding to the target object 20 (empty circles). may include In this case, if the target lidar points 2a, 2b, and 2c are accumulated without considering the movement of the target object 20, the accumulated target lidar points cannot accurately model the target object 20. . More specifically, referring to (b) of FIG. 2 , target lidar points (hatched circles) acquired at a previous point in time do not accurately model the target object 20 . Accordingly, as shown in (c) of FIG. 2 , the target lidar points acquired at a previous time (hatched circles) should be accumulated in consideration of the movement of the target object 20 so that the accumulated target lidar points are not generated by the target object 20 ) can be accurately modeled.

Hereinafter, a modeling system will be described.

3 is a diagram of a modeling system according to an embodiment. Referring to FIG. 3 , a modeling system according to an embodiment may include a sensor device 100 , a modeling device 200 , an output device 300 , a server 400 , and a user terminal 500 .

The sensor device 100 may detect information about the target object. Examples of the sensor device 100 may include a lidar module, a camera module, and the like, but is not limited thereto.

When the sensor device 100 includes a lidar module, the sensor device 100 may generate a lidar point cloud. The sensor device 100 may scan a target object to generate a lidar point cloud including one or more target lidar points.

When the sensor device 100 includes a camera module, the sensor device 100 may generate an image frame or a video. The sensor device 100 may generate an image frame or video including the target object by photographing the target object. Examples of the camera module may include, but are not limited to, an RGB camera, a thermal imaging camera, night vision, and the like.

The modeling apparatus 200 may model the target object. The modeling apparatus 200 may model the target object based on the lidar point cloud. The modeling apparatus 200 may determine one or more target lidar points from among lidar point clouds. The modeling apparatus 200 may determine one or more target LiDAR points based on an image frame or a video. The modeling apparatus 200 may model the target object by accumulating one or more target LIDAR points.

The modeling apparatus 200 may acquire a lidar point cloud or data related thereto. For example, the modeling apparatus 200 may obtain a lidar point cloud or data related thereto from the sensor apparatus 100 including a lidar module.

The modeling apparatus 200 may acquire an image frame, a video, or data related thereto. For example, the modeling apparatus 200 may obtain an image frame, a video, or data related thereto from the sensor apparatus 100 including a camera module.

The modeling apparatus 200 may transmit the modeling result to at least one of the output apparatus 300 and the server 400 .

The output device 300 may output the modeling result received from the modeling device 200 . The output of the modeling result is not limited to visual or auditory output, but may be formed in various forms.

The output device 300 may be implemented as a display that outputs an image, a speaker that outputs a sound, a haptic device that generates vibration, and/or other various types of output means. Alternatively, the output device 300 may be implemented in the form of an output interface (USB port, PS/2 port, etc.) instead of a device that outputs information to the outside by itself.

The server 400 may manage the modeling result received from the modeling apparatus 200 . The server 400 may transmit the modeling result received from the modeling apparatus 200 to the user terminal 500 . Examples of the user terminal 500 include, but are not limited to, a smartphone, a tablet, and a PC. The server 400 may receive a user's request from the user terminal 500 and provide corresponding information to the user through the user terminal 500 . For example, when receiving a user's modeling result request from the user terminal 500 , the server 400 may provide the modeling result to the user through the user terminal 500 .

Not all of the components shown in FIG. 3 are essential components of the modeling system. The modeling system may further include components not shown in FIG. 3 . In addition, at least some of the components of the modeling system shown in FIG. 3 may be omitted.

Also, although the modeling apparatus 200 and the server 400 are described as separate components in FIG. 3 , the modeling apparatus 200 and the server 400 may be the same single component.

Hereinafter, the modeling apparatus will be described in more detail.

4 is a diagram of a modeling apparatus according to an exemplary embodiment. The modeling apparatus may perform a modeling method of modeling a target object, which will be described later.

Referring to FIG. 4 , the modeling apparatus 200 according to an embodiment may include a communication unit 210 , a storage unit 220 , and a control unit 230 .

The communication unit 210 may communicate with various types of external devices according to various types of communication methods. The communication unit 210 may perform bi-directional or unidirectional communication. The communication unit 210 may perform communication by wire or wirelessly. Examples of the communication unit 210 may include a transceiver, a cellular module, a WiFi module, a Bluetooth module, a GNSS module, an NFC module, an RF module, a Zigbee module, and the like, but is not limited thereto.

The communication unit 210 may receive the lidar point cloud or data related thereto from the outside. For example, the communication unit 210 may receive a lidar point cloud or data related thereto from the sensor device 100 .

The communication unit 210 may receive an image frame, a video, or data related thereto from the outside. For example, the communication unit 210 may receive an image frame, a video, or data related thereto from the sensor device 100 .

The communication unit 210 may transmit the modeling result to at least one of the output device 300 and the server 400 .

The storage 220 may store various data and programs necessary for the modeling apparatus 200 to operate. The storage 220 may store data obtained by the modeling apparatus 200 . For example, the storage unit 220 may store a lidar point cloud, a target lidar point, or data related thereto. As another example, the storage 220 may store an image frame, a video, or data related thereto. Examples of the storage unit 220 include non-volatile semiconductor memory, hard disk, flash memory, random access memory (RAM), read only memory (ROM), electrically erasable programmable read-only memory (EEPROM), or other tangible (tangible) memory. ) may be a non-volatile recording medium, but is not limited thereto.

The controller 230 may process and calculate various types of information in the modeling apparatus 200 . The controller 230 may control other components constituting the modeling apparatus 200 .

The controller 230 may be implemented as a computer or a similar device according to hardware, software, or a combination thereof. In terms of hardware, the controller 230 may be one or a plurality of processors. Alternatively, the control unit 230 may be provided to processors that are physically spaced apart and cooperate through communication. Examples of the control unit 230 include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing unit (DSP), a state machine, and an application-specific semiconductor. (Application Specific Integrated Circuit, ASIC), may be a radio-frequency integrated circuit (Radio-Frequency Integrated Circuit, RFIC), but is not limited thereto. The software control unit 230 may be provided in the form of a program for driving the hardware control unit 230 .

The controller 230 may model the target object. The controller 230 may model the target object based on the lidar point cloud. The controller 230 may determine one or more target lidar points from among the lidar point clouds. The controller 230 may determine one or more target LiDAR points based on an image frame or a video. The controller 230 may model the target object by accumulating one or more target LIDAR points.

The controller 230 may control the communication unit 210 to receive the lidar point cloud or data related thereto.

The controller 230 may control the communication unit 210 to receive an image frame, a video, or data related thereto.

The controller 230 may control the communication unit 210 to transmit the modeling result to at least one of the output device 300 and the server 400 .

The controller 230 may control the storage 220 to store various data and programs necessary for the modeling apparatus 200 to operate. The control unit 230 may control to store the data acquired through the communication unit 210 in the storage unit 220 .

Hereinafter, unless otherwise specified, an operation performed by the modeling apparatus 200 may be interpreted as being performed by the controller 230 or performed by the controller 230 controlling other components of the modeling apparatus 200 . .

Not all of the components shown in FIG. 4 are essential components of the modeling apparatus 200 . The modeling apparatus 200 may further include components not shown in FIG. 4 . In addition, at least some of the components of the modeling apparatus 200 shown in FIG. 4 may be omitted.

Hereinafter, the modeling method will be described in more detail.

In the present application, it is assumed that the modeling method according to an embodiment is performed by the modeling apparatus, but this is only described as an example of various embodiments, and does not mean that the modeling method should be performed only by the modeling apparatus . For example, a part of the modeling method may be performed in a modeling device, and the rest may be performed in a server.

According to an embodiment, the modeling apparatus may model the target object by accumulating one or more target lidar points corresponding to the target object for a predetermined time period. In this case, a target lidar point (hereinafter referred to as a “first target lidar point”) corresponding to the first time point within the predetermined time period and a second time point within the predetermined time period - the second time point is the The target object may be modeled using a target lidar point (hereinafter, referred to as a “second target lidar point”) corresponding to -.

1. Modeling method using conversion information calculated based on the image frame

5 is a diagram illustrating a modeling method using transformation information calculated based on an image frame according to an exemplary embodiment.

According to an embodiment, the modeling method may include acquiring an image frame. For example, the modeling apparatus may acquire an image frame. The image frame may be generated from a camera module. The modeling apparatus may acquire a plurality of image frames. The plurality of image frames include an image frame corresponding to a first time point (hereinafter referred to as a “first image frame”) and an image frame corresponding to a second time point subsequent to the first time point (hereinafter referred to as a “second image frame”). ) may be included.

The position of a pixel or object within an image frame may be expressed in image coordinates. The image coordinates may be two-dimensional coordinates.

6 is a diagram of image coordinates according to an exemplary embodiment. A position of a specific pixel in an image frame may be expressed on an xy image coordinate system in which a reference pixel is defined as an origin. For example, referring to FIG. 6 , the coordinates of a specific pixel may be determined based on the upper left pixel 3a. In this case, the image coordinates of the upper left pixel 3a of FIG. 6 may be (0,0), and the image coordinates of the specific pixel 3b may be expressed as (5,3).

According to one embodiment, the modeling method may include obtaining one or more target lidar points. For example, the modeling apparatus may acquire one or more target lidar points. The target LIDAR points may be generated from the LIDAR module. The target lidar points are one or more target lidar points corresponding to a first viewpoint (hereinafter referred to as “first target lidar points”) and one or more targets corresponding to a second viewpoint that is after the first viewpoint. It may include lidar points (hereinafter referred to as “second target lidar points”).

7 is a diagram of a target lidar point according to an embodiment, and is a conceptual diagram of a lidar point cloud obtained near a port.

The lidar module may create lidar point clouds. Referring to FIG. 7 , the lidar point cloud may include lidar points 4a corresponding to the ship, lidar points 4b corresponding to the quay wall, and lidar points 4c corresponding to the sea level. have.

As described above, the target lidar point corresponds to the target object in the lidar point cloud. Accordingly, when the target object is a vessel, the target lidar point is the lidar points 4a corresponding to the vessel in the lidar point cloud.

According to an embodiment, the modeling method may include obtaining a lidar point cloud. For example, the modeling device may acquire a lidar point cloud. The lidar point cloud may include a first lidar point cloud corresponding to a first viewpoint and a second lidar point cloud corresponding to a second viewpoint.

According to an embodiment, the modeling method may include determining a target lidar point from among lidar point clouds. For example, the modeling apparatus may determine a target lidar point from among lidar point clouds. Referring to FIG. 7 , the modeling apparatus is applied to a ship as a target object among lidar points 4a corresponding to the ship, lidar points 4b corresponding to the quay wall, and lidar points 4c corresponding to the sea level. Corresponding lidar points 4a may be determined as target lidar points. A more detailed description of determining a target lidar point from among lidar point clouds will be described later.

Referring back to FIG. 5 , a modeling method according to an embodiment may include acquiring a first image frame. For example, in step S1100, the modeling apparatus may acquire a first image frame. A modeling method according to an embodiment may include acquiring one or more first target LIDAR points. For example, in step S1100, the modeling apparatus may acquire one or more first target LIDAR points. In this case, the first image frame and the first target lidar points may correspond to the same viewpoint.

A modeling method according to an embodiment may include acquiring a second image frame. For example, in step S1200 , the modeling apparatus may acquire the second image frame. A modeling method according to an embodiment may include acquiring one or more second target LiDAR points. For example, in step S1200, the modeling apparatus may acquire one or more second target LiDAR points. In this case, the second image frame and the second target lidar points may correspond to the same viewpoint.

According to an embodiment, the modeling method may include calculating transformation information based on the first image frame and the second image frame ( S1300 ). For example, the modeling apparatus may calculate transformation information based on the first image frame and the second image frame. The transformation information may reflect the movement of the target object. For example, the transformation information may reflect the movement of the target object between the first viewpoint and the second viewpoint. The transformation information may be applied to image coordinates. For example, the modeling apparatus may calculate image coordinates corresponding to the second viewpoint by applying the transformation information to the image coordinates corresponding to the first viewpoint.

FIG. 8 is a view for explaining conversion information according to an exemplary embodiment, in which (a) of FIG. 8 is a first image frame 50a corresponding to a first view, and (b) is a second image corresponding to a second view. 2 is an image frame 50b, and the target object is a ship.

The modeling apparatus may obtain a region corresponding to the target object of the second image frame by applying transformation information to the region corresponding to the target object of the first image frame 50a.

In terms of pixels, the modeling apparatus applies transformation information to the image coordinates of pixels (hereinafter referred to as “first pixels” 5a, 5b, 5c, 5d) corresponding to the target object of the first image frame 50a. Thus, image coordinates of pixels corresponding to the target object of the second image frame (hereinafter referred to as “second pixels”, 5a', 5b', 5c', 5d') may be obtained. In this case, the first pixels 5a, 5b, 5c, and 5d may correspond to the second pixels 5a', 5b', 5c', and 5d', respectively. That is, pixels corresponding to each other among the first pixels 5a, 5b, 5c, and 5d and the second pixels 5a', 5b', 5c', and 5d' may correspond to the same portion of the target object.

The following [Equation 1] relates to a homography matrix, which is an example of transformation information.

[Equation 1]

Here, (x,y) and (x',y') denote image coordinates of different viewpoints, and the 3x3 matrix on the right side denotes a homography matrix. In this case, the image coordinates (x',y') of a viewpoint different from the specific viewpoint may be calculated by applying the homography matrix to the image coordinates (x,y) of the specific viewpoint.

The modeling apparatus may obtain transformation information based on characteristic information (hereinafter, referred to as “target characteristic information”) related to the target object obtained from the image frame. The target characteristic information may reflect the shape of the target object. For example, the target characteristic information may reflect the shape of the target object included in the image frame. The feature information may have image coordinates corresponding thereto. Examples of the characteristic information include, but are not limited to, a characteristic point, a corner, an edge, and the like.

The modeling apparatus may determine target characteristic information among the characteristic information obtained from the image frame. More specific details on this will be described later.

9 is a diagram for obtaining conversion information according to an embodiment.

According to an embodiment, the modeling method may include acquiring target feature information from the first image frame ( S1310 ). For example, the modeling apparatus may obtain target feature information from the first image frame. Hereinafter, target characteristic information obtained from the first image frame is referred to as first target characteristic information. According to an embodiment, the modeling method may include obtaining target feature information from the second image frame (S1320). For example, the modeling apparatus may obtain target feature information from the second image frame. Hereinafter, target characteristic information obtained from the second image frame is referred to as second target characteristic information.

According to an embodiment, the modeling method may include matching the first target characteristic information and the second target characteristic information ( S1330 ). For example, the modeling apparatus may match the first target characteristic information and the second target characteristic information. In this case, the first target characteristic information and the second target characteristic information matching each other may correspond to the same part of the target object or may be predicted to correspond to the same part. Alternatively, the first target characteristic information and the second target characteristic information that match each other may have a minimum distance on the image frame.

According to an embodiment, the modeling method may include calculating transformation information based on the matched target feature information ( S1340 ). For example, the modeling apparatus may calculate transformation information based on the matched target characteristic information.

According to an embodiment, the modeling method may include converting image coordinates of the first target characteristic information by applying the transformation information to the first target characteristic information ( S1350 ). For example, the modeling apparatus may convert the image coordinates of the first target characteristic information by applying the transformation information to the first target characteristic information.

According to an embodiment, the modeling method may include comparing the transformed first target characteristic information and the second target characteristic information ( S1360 ). For example, the modeling apparatus may compare the converted first target characteristic information and the second target characteristic information. Comparing the transformed first target characteristic information and the second target characteristic information may include comparing a distance between the transformed first target characteristic information and the second target characteristic information with a predetermined value. More specifically, comparing the transformed first target characteristic information and the second target characteristic information is comparing the transformed first target characteristic information and the second target characteristic information with a predetermined value and the distance on the image coordinate system. can

The modeling apparatus may repeatedly perform steps S1330 to S1360 until the comparison result satisfies a predetermined condition. That the comparison result satisfies the predetermined condition may mean that the distance between the converted first target characteristic information and the second target characteristic information is less than or equal to a predetermined value. More specifically, that the comparison result satisfies the predetermined condition may mean that the distances of the transformed first target characteristic information and the second target characteristic information in the image coordinate system are less than or equal to a predetermined value. At this time, the modeling apparatus repeatedly calculates the transformation information by repeatedly performing steps S1330 to S1360 (it may be regarded as updating the transformation information), and repeats the image coordinates of the first target characteristic information to transform (the first target). It may be seen as updating the image coordinates of the feature information).

10 is a diagram of examples of conversion information calculation algorithms. (a) is a conventional iterative closest point (ICP) algorithm, and (b) is a homography ICP algorithm. Comparing the homography ICP algorithm with the conventional ICP algorithm, in the homography ICP algorithm, a cost function is changed to a homography matrix solver. Accordingly, in the conventional ICP algorithm, only translation and rotation between the first target characteristic information and the second target characteristic information could be considered, but in the homography ICP algorithm, not only translation and rotation but also scaling are considered together, and transformation information is more generally can be calculated.

Referring back to FIG. 5 , the modeling method according to an embodiment may include transforming the coordinates of one or more first target LiDAR points based at least in part on the transformation information ( S1400 ). For example, the modeling apparatus may transform the coordinates of the one or more first target LiDAR points based at least in part on the transformation information. As a more specific example, the modeling apparatus converts the coordinates (hereinafter referred to as "first coordinates") corresponding to the first viewpoints of the first target lidar points to the coordinates corresponding to the second viewpoints (hereinafter, " referred to as "second coordinates").

11 is a diagram for converting first coordinates of a first target lidar point into second coordinates according to an embodiment. In FIG. 11 , a case in which the first coordinate and the second coordinate on the lidar coordinate system are transformed will be described, but the present invention is not limited thereto. That is, the first coordinate and the second coordinate may be coordinates in another coordinate system having the same dimension as the lidar coordinate system. In addition, in FIG. 11 , O _L denotes the origin of the lidar coordinate system, and X _L , Y _L and Z _L denote the x-axis, y-axis, and z-axis of the lidar coordinate system, respectively. O _c denotes the origin of the camera coordinate system, and X _c , Y _c , and Z _c denote the x-axis, y-axis, and z-axis of the camera coordinate system, respectively.

Referring to FIG. 11A , the modeling apparatus may project the first target lidar point 6a to the image plane 60 based on the calibration information. In this case, the modeling apparatus may acquire image coordinates (x,y) corresponding to the first coordinates (x _L ,y _L ,z _L ) of the first target lidar point 6a. At this time, since the dimension of the first coordinate (x _L ,y _L ,z _L ) is higher than the dimension of the image coordinate (x,y), the first coordinate (x _L ,y _L ,z _L ) is set to the image plane ( 60) to obtain one of the image coordinates (x,y).

The calibration information may include lidar-camera calibration information and camera calibration information.

The lidar-camera calibration information may reflect at least one of a relative position and direction between the camera module and the lidar module. The lidar-camera calibration information may include at least one of translation information reflecting a relative position between the camera module and the lidar module and rotation information reflecting a relative direction. The modeling apparatus uses the lidar-camera calibration information to convert the first coordinates (x _L , y _L , z _L ) on the lidar coordinate system of the first target lidar point 6a to the coordinates on the camera coordinate system (x _c , y _c ,z _c ).

The camera calibration information may reflect internal parameters of the camera module. The modeling apparatus may project the coordinates (x _c , y _c , z _c ) on the camera coordinate system onto the image plane by using the camera calibration information. In this case, the modeling apparatus may acquire image coordinates (x, y) corresponding to the first target lidar point 6a.

The following [Equation 2] relates to an example of calibration information.

[Equation 2]

Here, x, y are image coordinates, x _c , y _c , z _c are coordinates on the camera coordinate system of the first target lidar point, and x _L , y _L , z _L are coordinates on the lidar coordinate system of the first target lidar point. coordinates, the A matrix is an example of camera calibration information as shown in [Equation 3] below, and R and t are a rotation matrix that is an example of rotation information as shown in [Equation 4] and a translation matrix that is an example of translation information, respectively to be.

[Equation 3]

[Equation 4]

Here, f _x and f _y are effective focal lengths in horizontal and vertical directions, respectively, skew _c is a skew coefficient, and c _x and c _y are coordinates of principal points.

Accordingly, in Equation 2, a rotation matrix and a translation matrix [R|t] are applied to the coordinates (x _L , y _L , z _L ) of the first target lidar point 6a on the lidar coordinate system to apply the first target The coordinates (x _c ,y _c ,z _c ) on the camera coordinate system of the lidar point 6a are obtained, and the image coordinates (x,y) are obtained by applying the A matrix.

Referring to FIG. 11B , the modeling apparatus may transform image coordinates (x,y) corresponding to the first target lidar point 6a projected on the image plane 60 . The modeling apparatus may convert image coordinates (x,y) by reflecting the movement of the target object. For example, the modeling apparatus may convert the image coordinates (x,y) by reflecting the movement of the target object between the first viewpoint and the second viewpoint. The modeling apparatus may convert image coordinates based on the above-described transformation information. The modeling apparatus may obtain the transformed image coordinates (x',y') by transforming the image coordinates (x,y).

Referring to (c) of FIG. 11 , the modeling apparatus provides a second coordinate (x) on the lidar coordinate system of the first target lidar point 6a from the image coordinates (x',y') converted based on the calibration information. At least one of _L ',y _L ',z _L ') and coordinates on the camera coordinate system (x _c ',y _c ',z _c ') may be generated. In this case, the second coordinates (x _L ', y _L ', z _L ') and the coordinates on the camera coordinate system (x _c ', y _c ', z _c ) rather than the dimensions of the transformed image coordinates (x',y') Since the dimension of ') is high, a group of coordinates including a plurality of coordinates may be mapped to the transformed image coordinates (x',y'). _{( refer to Equation 2 )} should be selected as

The modeling apparatus may select a specific coordinate from among a plurality of coordinates as the second coordinate (x _L ',y _L ',z _L ') based on the first coordinate (x _L ,y _L ,z _L ). For example, the modeling apparatus may select the second coordinates so that the height values of the first coordinates and the second coordinates are the same. As a more specific example, when the X _L -Y _L plane in the LIDAR coordinate system is parallel to the ground or sea level and Z _L corresponds to the height from the ground or sea level, the transformed image coordinates (x',y') A coordinate in which z _L '=z _L among a plurality of coordinates may be selected as the second coordinate. In this case, since the height values of the first coordinates and the second coordinates of the first target lidar point 6a are the same, the target object is located on a predetermined plane (eg, the ground or sea level) between the first and second viewpoints. It can be seen that it is assumed that it is moving and that the second coordinate is selected. That is, in a sea situation, it can be seen that the ship moves along the sea level on the sea level between the first time point and the second time point, and in a road condition, it can be seen that the vehicle moves along the ground on the ground. Similarly, the modeling apparatus converts a specific coordinate among a plurality of coordinates based on the coordinates (x _c ,y _c ,z _c ) on the camera coordinate system of the first viewpoint to the coordinates (x _c ',y _c ) on the camera coordinate system of the second viewpoint. ',z _c ') can be selected. A redundant description thereof will be omitted.

Referring back to FIG. 5 , the modeling method according to an embodiment may include modeling the target object using the converted one or more first target lidar points and one or more second target lidar points ( S1500 ). have. For example, the modeling apparatus may model the target object using the converted one or more first target lidar points and one or more second target lidar points.

The modeling apparatus may model the target object by accumulating target lidar points for a predetermined time period. In this case, the first viewpoint corresponding to the first target lidar points and the second viewpoint corresponding to the second target lidar points may be included in the predetermined time period.

12 and 13 are diagrams for explaining a time period in which target LiDAR points are accumulated, according to an embodiment.

The modeling apparatus may model the target object by accumulating target lidar points corresponding to a time interval between the reference viewpoint and the current viewpoint. Referring to FIG. 12A , the modeling apparatus may model the target object by accumulating target LiDAR points corresponding to the time interval Δt ₁₂ between the reference time t ₁ and the current time t ₂ . Referring to (b) of FIG. 12 , when time passes and the current time point changes from t ₂ to t ₃ , the modeling device is a target lid corresponding to the time interval Δt ₁₃ between the reference time point t ₁ and the current time point t ₃ . A target object may be modeled by accumulating points. In this case, the time period for accumulating the target lidar points may increase as time passes.

The modeling apparatus may model the target object by accumulating target LiDAR points corresponding to a time section between the current time point and a time point earlier than the current time point by a predetermined time. Referring to (a) of FIG. 13 , the modeling apparatus accumulates target LiDAR points corresponding to a time interval Δt ₄₅ between a current time t ₅ and a time t ₄ that is a predetermined time earlier therefrom to model the target object. can Referring to (b) of FIG. 13 , when time passes and the current time point changes from t ₅ to t ₇ , the modeling apparatus calculates the time interval Δt between the current time point t ₇ and the time point t ₆ that is a predetermined time earlier therefrom. The target object may be modeled by accumulating target lidar points corresponding to ₆₇ . In this case, the time interval for accumulating the target LiDAR points may not change even if time passes (Δt ₄₅ = Δt ₆₇ ).

Hereinafter, a method of determining a target lidar point among lidar point clouds will be described in more detail.

According to an embodiment, the modeling method may include determining a target lidar point from among lidar point clouds by using object type information. For example, the modeling apparatus may determine a target lidar point from among lidar point clouds by using object type information. The object type information may be generated based on an image frame. The object type information may reflect information on the type of object included in the image frame. The object type information may include data corresponding to a target object included in the image frame and indicating the target object. The object type information may reflect a location of the target object in the image frame.

14 is a diagram for describing object type information generated based on an image frame according to an exemplary embodiment. (a) of FIG. 14 is an image frame 70a captured by the camera module, and (b) is a segmentation image 70b that visualizes segmentation information, which is an example of object type information generated based on the image frame 70a. and (c) is a detection image 70c in which detection information, which is another example of object type information generated based on the image 70a, is visualized.

According to an embodiment, the modeling apparatus may generate segmentation information based on the image frame 70a. The segmentation information may include a plurality of segmentation data. Each of the plurality of segmentation data may correspond to each pixel of the image frame 70a. Each of the plurality of segmentation data may reflect the type of object represented by the corresponding pixel. Accordingly, the segmentation information may reflect the position of an object included in the image frame 70a or a pixel representing the object within the image frame 70a. The segmentation information may include at least one of segmentation data corresponding to a ship, segmentation data corresponding to a quay wall, segmentation data corresponding to the sea or sea level, segmentation data corresponding to a terrain, and segmentation data corresponding to the sky.

The segmentation information may be visualized. For example, the segmentation information may be visualized in the form of a segmentation image 70b expressed to be distinguished by a separate color or the like according to the type of object reflected by the plurality of segmentation data of the segmentation information. In this case, referring to FIG. 14(b), the segmentation image 70b is a region 7a corresponding to a ship, an area 7b corresponding to a quay wall, an area 7c corresponding to the sea or sea level, and the terrain. It may include a corresponding area 7d and an area 7e corresponding to the sky.

According to another embodiment, the modeling apparatus may generate detection information based on the image frame 70a. The detection information may include one or more detection data. Each of the one or more detection data may correspond to an object included in the image frame 70a. Each of the one or more detection data may reflect a type of an object corresponding thereto. Each of the one or more detection data may reflect a position of a corresponding object within the image frame 70a. The detection information may include at least one of detection data corresponding to a ship, detection data corresponding to a quay wall, detection data corresponding to the sea or sea level, detection data corresponding to a terrain, and detection data corresponding to the sky.

The detection information may be visualized. For example, the detection information may be visualized in the form of a detection image 70c in which a position in an image frame 70a of an object reflected by one or more detection data of the detection information is expressed in the form of a bounding box. In this case, referring to FIG. 14C , the detection image 70c may include a region 7f corresponding to the vessel.

The modeling apparatus may generate object type information using an artificial neural network. Examples of the artificial neural network include, but are not limited to, a convolution neural network (CNN), you only look once (YOLO), and a single shot multibox detector (SSD). The artificial neural network may be learned through various methods such as supervised learning, unsupervised learning, reinforcement learning, and imitation learning. Also, the modeling apparatus does not necessarily need to generate the object type information using the artificial neural network, and may generate the object type information by other methods.

The modeling apparatus may identify the type of object corresponding to one or more lidar points included in the lidar point cloud by using the object type information. For example, the modeling apparatus projects the lidar point cloud on an image plane, and a point at which each of one or more lidar points included in the lidar point cloud is projected on the image plane corresponds to a certain type of object. It is possible to identify the type of object corresponding to each of the lidar points by determining whether or not the object type is determined using the object type information.

The modeling apparatus may determine one or more target LiDAR points from among the LiDAR point clouds based on the identified result. For example, the modeling apparatus may determine one or more lidar points identified as corresponding to the target object in the lidar point cloud as one or more target lidar points.

15 is a diagram for determining a target LIDAR point from among LIDAR point clouds using object type information according to an exemplary embodiment, when a vessel is a target object.

Referring to FIG. 15A , the modeling apparatus may acquire a lidar point cloud 8a.

Referring to FIG. 15B , the modeling apparatus may project the lidar point cloud 8a onto the image plane 80a. In this case, the modeling device may acquire the projected lidar point cloud 8b.

Referring to FIG. 15C , the modeling apparatus may determine one or more target LiDAR points from among the projected LiDAR point clouds 8b using segmentation information. The modeling apparatus may identify the type of object corresponding to the projected lidar point cloud 8b using the segmentation information. For example, the projected lidar point cloud 8b may include one or more lidar points 8c corresponding to a ship as a target object and one or more lidar points 8d corresponding to sea or sea level. have. The modeling apparatus may determine one or more lidar points 8c corresponding to the target object as one or more target lidar points 8c based on the identified result. For example, the modeling device may include one or more lidar points 8c corresponding to the vessel and one or more lidar points 8c corresponding to the vessel among the one or more lidar points 8d corresponding to the sea or sea level. may be determined as one or more target lidar points 8c.

Referring to (d) of FIG. 15 , the modeling apparatus may determine one or more target lidar points from among the projected lidar point clouds 8b using detection information. The modeling apparatus may identify the type of object corresponding to the projected lidar point cloud 8b by using the detection information. The modeling apparatus may determine one or more lidar points 8c and 8d corresponding to the target object as one or more target lidar points 8c and 8d based on the identified result. Compared to the case of using the segmentation information of FIG. 15(c), in the case of segmentation information, objects included in an image frame are divided by pixels, whereas in the case of detection information, objects are classified in the form of a bounding box, so it actually corresponds to the target object. A lidar point 8d that does not do so may also be determined as a target lidar point. Accordingly, when the modeling apparatus uses segmentation information, the modeling apparatus may determine the target lidar point more accurately than using the detection information.

Hereinafter, a method of determining target characteristic information among the characteristic information obtained from the image frame will be described in more detail.

The modeling apparatus may determine target characteristic information among the characteristic information by using the above-described object type information. The modeling apparatus may identify the type of object corresponding to the feature information by using the object type information. The modeling apparatus may determine target characteristic information among the characteristic information based on the identified result. For example, the modeling apparatus may determine, as the target characteristic information, that which is identified as corresponding to the target object among the characteristic information.

16 is a diagram illustrating obtaining target characteristic information from characteristic information using object type information according to an exemplary embodiment, wherein a vessel is a target object.

Referring to FIG. 16A , the modeling apparatus may acquire an image frame 90a.

Referring to FIG. 16B , the modeling apparatus may acquire the feature information 9a from the image frame 90a.

Referring to FIG. 16C , the modeling apparatus may determine target characteristic information among the characteristic information 9a by using the segmentation information. The modeling apparatus may identify the type of object corresponding to the feature information 9a by using the segmentation information. For example, the characteristic information 9a may include characteristic information 9b corresponding to a ship as a target object and characteristic information 9c corresponding to the sea or sea level. The modeling apparatus may determine the characteristic information 9b corresponding to the target object as the target characteristic information 9b based on the identified result. For example, the modeling apparatus may determine the characteristic information 9b corresponding to the vessel among the characteristic information 9b corresponding to the vessel and the characteristic information 9c corresponding to the sea or sea level as the target characteristic information 9b.

Referring to FIG. 16D , the modeling apparatus may determine target characteristic information among the characteristic information 9a by using the detection information. The modeling apparatus may identify the type of object corresponding to the feature information 9a by using the detection information. The modeling apparatus may determine the characteristic information 9b and 9c corresponding to the target object as the target characteristic information 9b and 9c based on the identified result. Compared with the case of using the segmentation information of FIG. 16(c), in the case of segmentation information, objects included in an image frame are divided by pixels, whereas in the case of detection information, objects are classified in the form of a bounding box, so it actually corresponds to the target object. The feature information 9c that is not used may also be determined as the target feature information. Accordingly, when the modeling apparatus uses the segmentation information, the modeling apparatus may determine the target characteristic information more accurately than using the detection information.

The modeling apparatus may model the target object based on the voxelized target lidar point.

17 is a diagram related to voxelization according to an embodiment.

Referring to FIG. 17 , the 3D space may be divided into a plurality of voxel grids 9d through voxelization. One or more target lidar points 9e included in each voxel grid 9d may be treated as one representative lidar point 9f. Accordingly, the number of lidar points to be handled by the modeling apparatus is reduced, and thus, a computational cost required for the modeling apparatus to model the target object is reduced.

The coordinates of the representative lidar point 9f may be coordinates of any point within the voxel grid 9d. For example, the coordinates of the representative lidar point 9f may be coordinates of the center point of the voxel grid 9d, but is not limited thereto.

2. Modeling method using conversion information calculated based on lidar points

In the above, a modeling method using conversion information calculated based on an image frame has been described. Hereinafter, the modeling method using the transformation information calculated based on the lidar point will be mainly described with respect to the difference from the modeling method using the transformation information calculated based on the table of contents 1. The image frame. Also, the contents described in Table of Contents 1. A modeling method using transformation information calculated based on an image frame can be applied to a modeling method using transformation information calculated based on a lidar point.

18 is a diagram illustrating a modeling method using transformation information calculated based on a lidar point according to an embodiment.

According to an embodiment, the modeling method may include acquiring first target lidar points and second target lidar points ( S2100 ). For example, the modeling apparatus may acquire first target LIDAR points and second target LIDAR points. For the acquisition of the first target LIDAR points and the second target LIDAR points, the contents described in Table of Contents 1. Modeling method using transformation information calculated based on an image frame may be applied, and thus a redundant description will be omitted.

According to an embodiment, the modeling method may include generating a plurality of planes using the second target lidar points ( S2200 ). For example, the modeling apparatus may generate a plurality of planes using the second target lidar points.

19 is a diagram for generating planes using lidar points according to an embodiment.

The modeling apparatus may generate planes related to each of the second target LiDAR points for each of the second target LiDAR points. Hereinafter, each of the second target LiDAR points is referred to as a target point. Accordingly, the modeling apparatus generates planes related thereto by using one of the second target lidar points as a target point, and generates planes related thereto again using the other one of the second target lidar points as a target point, , this may be repeatedly performed for all of the second target LIDAR points to generate planes related thereto for each of the second target LIDAR points.

The modeling apparatus may generate planes related to the target point by using the target point and LiDAR points adjacent thereto. The target point and LiDAR points adjacent thereto may be included in the second target LiDAR points. Referring to FIG. 19 , planes Π ₁ , Π ₂ , Π ₃ , Π using the target point p ₀ and LiDAR points p ₁ , p ₂ , p ₃ , and p ₄ adjacent thereto for the target point p ₀ . ₄ can be created. Similarly, when the target point is p ₂ , planes may be generated using LiDAR points p ₀ , p ₅ , p ₆ , and p ₇ adjacent thereto.

19 shows that planes are generated using four LiDAR points adjacent to the target point, but the number of adjacent LiDAR points may be different.

In addition, in FIG. 19 , the lidar is a rotating lidar, wherein lidar points adjacent to the target point include lidar points adjacent to the target point left and right and lidar points vertically adjacent to the target point, and the left and right lidar points LiDAR points adjacent to are LiDAR points that exist in the same ring as the target point and whose order is immediately before and after the target point, and the upper and lower adjacent LiDAR points are in rings above and below the target point and the distance is Although illustrated as being the closest lidar points, lidar points adjacent to the target point are not necessarily determined in this way. For example, both the target point and the adjacent LIDAR points are based on the distance from the target point (eg, a predetermined number of LIDAR points that are close to the target point are determined as LIDAR points adjacent to the target point) may be determined, or may be determined in another way.

In some embodiments, the modeling apparatus may merge at least some of the planes related to the target point with each other. In this case, the modeling apparatus may merge the planes in consideration of the similarity between the planes. For example, referring to FIG. 19 , the modeling apparatus may calculate a distance d ₁₂ between the first plane Π ₁ and the second plane Π ₂ as shown in Equation 5 below.

[Equation 5]

Here, x ₀ , x ₁ and x ₂ are coordinates of p ₀ , p ₁ and p ₂ , respectively, and n ₁ and n ₂ are normal vectors of the first plane Π ₁ and the second plane Π ₂ , respectively. . In this case, when the calculated distance d ₁₂ is smaller than a threshold, the first plane and the second plane may be merged into one plane Π ₁₂ , for example, as in Equation 6 below.

[Equation 6]

Here, n ₁₂ is the normal vector of the merged plane Π ₁₂ . Similarly, the third plane Π ₃ and the fourth plane Π ₄ of FIG. 19 may be merged into one plane Π ₃₄ . Also, the merged planes Π ₁₂ and Π ₃₄ may similarly be merged into one plane Π ₁₂₃₄ .

As described above, since planes related to the target point may be merged with each other, the number of planes related to the target point may be different for each target point. For example, the number of planes associated with a first target point of the second target LiDAR points may be different from the number of planes associated with a second target point that is different from the first one of the second target LiDAR points. can

On the other hand, the planes related to the target point do not have to be merged in the same way as above. For example, the distance between the planes may be calculated using a method different from Equation 5, or a merged plane may be generated using a method different from Equation 6 above.

Referring back to FIG. 18 , the modeling method according to an embodiment is a first target lidar using transformation information calculated based on first target lidar points, second target lidar points, and a plurality of planes. It may include transforming the coordinates of the points (S2300). For example, the modeling apparatus may transform the coordinates of the first target lidar points using transformation information calculated based on the first target lidar points, the second target lidar points, and the plurality of planes. .

20 is a diagram illustrating coordinate transformation of lidar points according to an embodiment.

According to an embodiment, the modeling method may include determining a plane corresponding to each first target lidar point from among a plurality of planes for each of the first target lidar points ( S2310 ). For example, the modeling apparatus may determine, for each of the first target LiDAR points, a plane corresponding to each of the first target LiDAR points from among a plurality of planes. Here, the plurality of planes may be generated using the second target LiDAR points in step S2200. Hereinafter, each of the first target LiDAR points is referred to as a query point. Accordingly, the modeling apparatus uses one of the first target LiDAR points as a query point to determine a plane corresponding thereto, and uses the other one of the first target LiDAR points as a query point to select a plane corresponding thereto again. determined, and repeating this for all of the first target LiDAR points to determine a corresponding plane for each of the first target LiDAR points.

The modeling apparatus may determine a plane corresponding to the query point based on a distance between the query point and the plurality of planes. For example, the modeling apparatus may determine, among the plurality of planes, a plane having a minimum distance to the query point as a plane corresponding to the query point.

The modeling apparatus may determine a specific point corresponding to the query point among the second target LIDAR points, and determine a plane corresponding to the query point from among planes related to the specific point. For example, the modeling apparatus determines a specific point having a minimum distance to a query point from among the second target LiDAR points, and selects a plane having a minimum distance to the query point from among planes related to the specific point. It can be determined as a plane corresponding to the query point.

According to an embodiment, the modeling method may include calculating transformation information based on the first target LiDAR points and corresponding planes ( S2320 ). For example, the modeling apparatus may calculate transformation information based on the first target LIDAR points and corresponding planes. The transformation information may reflect the movement of the target object. For example, the transformation information may reflect the movement of the target object between the first viewpoint and the second viewpoint. The transformation information may be applied to 3D coordinates. For example, the modeling apparatus may calculate 3D coordinates corresponding to the second viewpoint of the LiDAR point by applying the transformation information to the 3D coordinates corresponding to the first viewpoint of the LIDAR point.

The modeling apparatus may calculate transformation information based on a distance between the first target LIDAR points and corresponding planes. For example, the modeling apparatus may calculate transformation information such that a distance between the first target LiDAR points and corresponding planes is minimized. The modeling apparatus calculates a distance between one of the first target LiDAR points and a corresponding plane, calculates a distance between the other one of the first target LiDAR points and a corresponding plane, and calculates the distance between the first target LiDAR points and a corresponding plane. A distance between planes corresponding to each of the first target LiDAR points may be calculated by repeatedly performing all one target LiDAR points, and conversion information may be calculated based on the distance between the corresponding planes.

The transformation information may include at least one of translation information and rotation information. Equation 7 below represents a translation matrix, which is an example of translation information, and Equation 8 below is a rotation matrix, which is an example of rotation information, which approximates a case with a small rotation angle.

[Equation 7]

Here, t ₁ , t ₂ , and t ₃ are parallel movements in the x-axis, y-axis, and z-axis directions, respectively.

[Equation 8]

Here, α, β and γ are rotation angles about the x-axis, y-axis, and z-axis, respectively.

According to an embodiment, the modeling method may include transforming the coordinates of the first target lidar points by applying the transformation information to the coordinates of the first target lidar points ( S2330 ). For example, the modeling apparatus may transform the coordinates of the first target lidar points by applying the transformation information to the coordinates of the first target lidar points.

According to an embodiment, the modeling method may include comparing the converted first target LIDAR points and the second target LIDAR points ( S2340 ). For example, the modeling apparatus may compare the converted first target LIDAR points and the second target LIDAR points. Comparing the transformed first target lidar points and the second target lidar points is comparing a distance between the transformed first target lidar points and the second target lidar points with a predetermined value. can More specifically, comparing the transformed first target lidar points and the second target lidar points means that the transformed first target lidar points and the second target lidar points have a three-dimensional coordinate system (eg, la It may be comparing the distance on the coordinate system with a predetermined value.

The modeling apparatus may repeatedly perform steps S2310 to S2340 until the comparison result satisfies a predetermined condition. That the comparison result satisfies the predetermined condition may mean that the distance between the converted first target lidar points and the second target lidar points is less than or equal to a predetermined value. More specifically, that the comparison result satisfies the predetermined condition means that the distances in the three-dimensional coordinate system (eg, the lidar coordinate system) of the transformed first target lidar points and the second target lidar points are predetermined values. It may mean the following. As the modeling apparatus repeatedly performs steps S2310 to S2340, it repeatedly calculates the transformation information (it may be seen as updating the transformation information), and repeats the coordinates of the first target LiDAR points and transforms them (the first target LA). It can also be seen as updating the coordinates of the points.)

According to an embodiment, the modeling method may include modeling the target object using the converted first target LIDAR points and the second target LIDAR points ( S2400 ). For example, the modeling apparatus may model the target object using the converted first target LIDAR points and the second target LIDAR points. For the modeling of the target object, since the contents described in Table of Contents 1. A modeling method using transformation information calculated based on an image frame may be applied, a redundant description will be omitted.

21 is a diagram of an example of an algorithm related to a modeling method using transformation information calculated based on a LiDAR point. (a) relates to the entire algorithm, (b) is a query point among the entire algorithm of (a). It relates to a detailed algorithm for determining the plane corresponding to . Here, the algorithm of Figure 21 (a) was named Switchable Plane ICP (SP-ICP). In Figure 21 (a), P and Q are target points and query points, respectively, xP, rP and sP are the coordinates of the target point, the ring corresponding to the target point, the sequence corresponding to the target point, respectively, xQ is Coordinates of the query point. R and t are rotation and translation matrices, respectively. SP-ICP generates a plurality of planes associated therewith for a specific target point (line 3), and determines a corresponding plane among the plurality of planes for a specific query point (lines 8, 9), wherein the specific query point It is an algorithm that includes determining the plane corresponding to the point in every iteration process.

22 is a table regarding the accuracy of SP-ICP, which is an embodiment of a modeling method, and is a KITTI benchmark test result.

22, for comparison, G-ICP, NDT, and point-to-plane ICP were also tested together, and all sequences 00-10 of the KITTI benchmark were tested. Only 3D LiDAR data was used for odometry measurement, and the previous matching results were used for initial posture. Input data and initial posture were taken from each algorithm. In all algorithms, the original cloud for the target points and the 20 cm voxelized cloud were used for the source points. For G-ICP, NDT and point-to-plane ICP, the default parameters of the PCL library were used. The normal vector for each point in the point-to-plane ICP was estimated as adjacent points within a radius of 20 cm based on each point. In order to test the sparse situation, since the KITTI benchmark data is too dense, we tested SP-ICP with extra normal added, and extra normal was added for point-to-plane ICP for fair comparison. SP-ICP without extra normal was also tested.

22 shows the translation and rotation errors for each algorithm. The translation error of the point-to-plane ICP was 3.93% and the rotation error was 0.0344 deg/m, the translation error of the NDT was 2.01% and the rotation error was 0.0164 deg/m, and the translation error of the G-ICP was 1.83 %, the rotation error was 0.0133 deg/m. On the other hand, the translation error of the SP-ICP was 1.28% and the rotation error was 0.0087 deg/m, and the translation error of the SP-ICP with extra normal added was 1.17% and the rotation error was 0.0082 deg/m. Through this, it was confirmed that when using the SP-ICP series, the accuracy was improved compared to other algorithms.

23 and 24 are diagrams for visualization of a modeling result using SP-ICP, which is an embodiment of a modeling method, and are test results for a situation in which a vessel is detected in a port. 23 is a three-dimensional view, FIG. 24 is a two-dimensional projected view, in each of FIGS. 23 and 24 (a) is SP-ICP result, (b) is G-ICP result, (c) is NDT result , the left figure shows the results for dataset 1, the middle figure shows the results for dataset 2, and the figure on the right shows the results for dataset 3.

23 and 24, for comparison, G-ICP and NDT were also tested together, and as an example of a situation in which the point density of the lidar point cloud is thinner than the KITTI benchmark data, a ship in a port, etc. The detection conditions were tested. The distance between the lidar and the ship is approximately 30m to 150m, which is suitable for testing the situation where the point density of the lidar point cloud is sparse because the distance between the lidar and the object increases compared to the situation of detecting a vehicle on the road. . The test was conducted using a 16-channel lidar (RS-LiDAR-16) and two cameras. Depending on the distance, 3 (approximately 100m) to 8 (approximately 40m) rings could reach the object, and the spacing between the rings was 2m to 4m depending on the distance, and between adjacent lidar points within the same ring. The spacing between the two was about 15 cm to 30 cm. The approximate size of the detected object, the vessel, was about 120m to 200m in length and 30m in width. In one scan, it was possible to acquire about 500 to 5000 LiDAR points depending on the distance.

The test was conducted on three datasets representing the berthing situation of the vessel, and the accuracy was compared with the visualized results because the ground truth could not be prepared. 100 frames of LiDAR data were accumulated, and the red-green-blue gradation of LiDAR points means the earliest LiDAR point (red) to the latest LiDAR point (blue) in time. The clearer the vessel can be seen, the higher the accuracy of the algorithm can be considered. When G-ICP and NDT were used, the vessel was modeled dirty or distorted for all datasets, whereas when SP-ICP was used, it was confirmed that the vessel was modeled more clearly and clearly than G-ICP and NDT.

25 is a more numerical expression of the accuracy of the test results of FIGS. 23 and 24 and shows the number of voxelized LiDAR points in two-dimensionally projected data. In general, the smaller the number of voxelized LiDAR points, the clearer and more clearly the object is modeled. Referring to FIG. 25 , it was found that the number of voxelized LiDAR points of SP-ICP was small for all datasets 1, 2, and 3, and thus SP compared to G-ICP and NDT. -It can be seen that the ICP modeled the object more accurately.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computing means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the present application has been described with reference to the examples above, the present application is not limited thereto, and it is apparent to those skilled in the art that various changes or modifications can be made within the spirit and scope of the present application, and thus It is pointed out that such changes or modifications fall within the scope of the appended claims.

100: sensor device
200: modeling device
210: communication unit
220: storage
230: control unit
300: output device
400: server
500: user terminal

Claims (19)

In the modeling method of modeling a target object performed by one or more processors,
acquiring first target lidar points corresponding to a first viewpoint;
acquiring second target lidar points corresponding to a second viewpoint different from the first viewpoint;
For each of the second target LiDAR points, generating a plurality of planes related to each of the second target LiDAR points by using the respective second target LiDAR points and the second target LiDAR points adjacent thereto step;
Transforming the coordinates of the first target lidar points using transformation information calculated based on the first target lidar points, the second target lidar points, and the plurality of planes - the transformation information reflects the movement between the first viewpoint and the second viewpoint of the target object, and the coordinates at the first viewpoint of the first target lidar points are converted to the coordinates at the second viewpoint by the transformation information. Converted- ; and
Modeling the target object using the converted first target lidar points and the second target lidar points,
The step of transforming the coordinates is
determining, for each of the first target lidar points, a corresponding plane of the plurality of planes having a minimum distance to each of the first target lidar points;
calculating the transformation information based on the first target lidar points and the corresponding plane; and
and repeating the transforming of the coordinates of the first target lidar points by applying the transformation information to the coordinates of the first target lidar points until a predetermined condition is satisfied.
modeling method.
The method of claim 1,
The second target lidar points include a first target point and a second target point,
The plurality of planes are
a plurality of first planes generated using the first target point and the second target LiDAR points adjacent thereto and related to the first target point; and
The second target point and the second target LiDAR points adjacent thereto are generated using the second target point and include a plurality of second planes related to the second target point.
modeling method.
3. The method of claim 2,
Creating the plurality of planes comprises:
merging at least some of the plurality of first planes into one plane in consideration of the similarity between the plurality of first planes
modeling method.
3. The method of claim 2,
The step of determining the corresponding plane comprises:
determining, for each of the first target LiDAR points, a corresponding point, among the second target LiDAR points, having a minimum distance to each of the first target LiDAR points; and
determining the corresponding plane from among one or more planes related to the corresponding point having a minimum distance to each of the first target LiDAR points;
modeling method.
The method of claim 1,
Acquiring the first target LiDAR points includes:
obtaining a first image frame corresponding to the first viewpoint;
obtaining a first lidar point cloud corresponding to the first time point;
generating first object type information reflecting a first target area related to the target object in the first image frame; and
determining one or more lidar points corresponding to the first target area among the first lidar point clouds as the first target lidar points by using the first object type information
modeling method.
The method of claim 1,
The first time point and the second time point are characterized in that included in a predetermined time interval
modeling method.
A computer-readable recording medium in which a computer program for executing the method according to any one of claims 1 to 6 is recorded on a computer.
In the modeling system for modeling a target object,
a lidar module that generates a first lidar point cloud corresponding to a first viewpoint and a second lidar point cloud corresponding to a second viewpoint different from the first viewpoint; and
a modeling device for modeling the target object using the first lidar point cloud and the second lidar point cloud;
The modeling device is
obtaining the first lidar point cloud and the second lidar point cloud from the lidar module;
obtaining first target lidar points related to the target object from the first lidar point cloud;
obtaining second target lidar points related to the target object from the second lidar point cloud;
For each of the second target LiDAR points, generating a plurality of planes associated with each of the second target LiDAR points using the respective second target LiDAR points and the second target LiDAR points adjacent thereto; ,
Transform the coordinates of the first target lidar points using transformation information calculated based on the first target lidar points, the second target lidar points, and the plurality of planes, and the transformation information is It reflects the movement between the first and second viewpoints of the target object, and the coordinates of the first target LiDAR points at the first viewpoint are converted into coordinates at the second viewpoint by the transformation information. become- ,
modeling the target object using the converted first target lidar points and the second target lidar points;
To transform the coordinates,
For each of the first target LiDAR points, determine a corresponding plane of the plurality of planes having a minimum distance to each of the first target LiDAR points;
calculating the transformation information based on the first target lidar points and the corresponding plane;
Repeatedly performing transformation of the coordinates of the first target lidar points by applying the transformation information to the coordinates of the first target lidar points until a predetermined condition is satisfied
modeling system.
9. The method of claim 8,
Further comprising a camera module that generates a first image frame corresponding to the first viewpoint,
The modeling device is
obtaining the first image frame from the camera module;
In obtaining the first target LIDAR points related to the target object from the first LIDAR point cloud, first object type information reflecting the first target area related to the target object in the first image frame is obtained. and determining one or more lidar points corresponding to the first target area among the first lidar point clouds as the first target lidar points by using the first object type information.
modeling system. delete delete delete delete delete delete delete delete delete delete

Images