KR20230038067A - Method and apparatus for estimating position of a moving object - Google Patents

Abstract

This electronic device is designed to feature point information in the form of a landmark-based probability map from surrounding images acquired with an imaging device mounted on a moving object; obtain landmark-based 3D feature point information from high definition map data around the moving object; convert one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high definition map data into another dimension; calculate the likelihood between the feature point information of the surrounding image and the feature point information of the high definition map data; and estimate the location of the moving object based on the likelihood.

Description

Method and apparatus for estimating the position of a moving object

The following embodiments relate to a method and apparatus for estimating the position of a moving object.

Autonomous driving refers to driving to a given destination by recognizing the surrounding environment, determining the driving situation, and controlling the vehicle without the intervention of the driver. Recently, as small mobility devices such as vehicles as well as mobile robots are used for smart delivery and security, the demand for autonomous vehicles or mobile bodies supporting advanced driver assistance systems (ADAS) is increasing.

A key factor in autonomous driving is estimating one's own location, and various technologies can be used to estimate the location. For example, using a device such as LiDAR or a camera, create a map in advance and estimate a location on the map using sensor data measured while driving, or estimate an absolute location by receiving satellite signals using a precise GPS device. method is mainly used.

An operating method of an electronic device according to an embodiment includes obtaining 2D feature point information in the form of a landmark-based probability map from a surrounding image obtained by a photographing device mounted on a moving object; obtaining landmark-based 3D feature point information from high definition map data around the moving object; converting one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data into another dimension; calculating a likelihood between feature point information of the surrounding image and feature point information of the high-precision map data; and estimating the position of the moving object based on the degree of correspondence.

According to an embodiment, the 2D feature point information of the surrounding image may be obtained according to landmarks based on deep neural network (DNN)-based semantic segmentation.

According to an embodiment, the obtaining of the 3D feature point information from high definition map data around the moving object may include determining a landmark around the moving object from a high-definition map database based on location information of the moving object. Receiving 3D feature point information on a world domain; and converting the 3D feature point information on the world domain into a local domain for the photographing device.

According to an embodiment, the step of converting one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data into another dimension is based on inverse perspective mapping. and converting the 2D feature point information of the surrounding image into a 3D probability map form.

According to an embodiment, the step of converting one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data into another dimension is based on perspective mapping, The method may include projecting 3D feature point information of the high-precision map data onto a 2D probability map obtained from the surrounding image.

According to an embodiment, calculating a likelihood between feature point information of the surrounding image and feature point information of the high-precision map data may include, in a probability map obtained from the surrounding image, feature point information of the high-precision map data. summing up probabilities corresponding to each landmark in ; and calculating the degree of correspondence by multiplying the summed probabilities corresponding to the respective landmarks.

According to an embodiment, the step of estimating the position of the moving object based on the degree of correspondence may include estimating the position of the moving object according to a particle filter or maximum likelihood (ML) optimization method based on the degree of correspondence. It may include updating the result.

According to an embodiment, the moving object may be an autonomous vehicle or a vehicle supporting an advanced driver assistance system (ADAS).

According to an embodiment, the landmark may include at least one of a white lane, a yellow lane, a crosswalk, a speed bump, a traffic light, and a traffic sign.

A method for estimating the position of a moving object based on a particle filter according to an embodiment includes a 2D feature point in the form of a landmark-based probability map from a surrounding image obtained by a photographing device mounted on the moving object. obtaining information; obtaining landmark-based 3D feature point information from high definition map data around the moving object; predicting a plurality of particle positions corresponding to candidate positions of the moving object; Projecting the 3D feature point information onto a probability map obtained from the surrounding image for each of the plurality of particle positions; calculating a likelihood between the 3D feature point information projected on the probability map and the 2D feature point information of the probability map, for each of the plurality of particle positions; and estimating the position of the moving object by rearranging the positions of the plurality of particles based on the degree of correspondence.

According to an embodiment, the 2D feature point information may be obtained according to landmarks based on deep neural network (DNN)-based semantic segmentation.

According to an embodiment, the obtaining of the 3D feature point information from high definition map data around the moving object may include determining a landmark around the moving object from a high-definition map database based on location information of the moving object. Receiving 3D feature point information on a world domain; and converting the 3D feature point information on the world domain into a local domain for the photographing device.

According to an embodiment, the step of estimating the positions of the plurality of particles includes predicting the positions of the plurality of particles based on particle position information relocated from a previous viewpoint and movement displacement of the moving object from the previous viewpoint. can do.

According to an embodiment, the 3D feature point information may be projected into a 2D probability map obtained from the surrounding image based on perspective mapping.

According to an embodiment, calculating a likelihood between 3D feature point information projected on the probability map and 2D feature point information of the probability map may include: in the probability map, the projected 3D feature point information. summing probabilities corresponding to each landmark of the information; and multiplying the summed probabilities corresponding to each landmark.

According to an embodiment, the step of estimating the position of the moving object by rearranging the positions of the plurality of particles based on the degree of correspondence may include setting a weight for each of the positions of the plurality of particles according to the degree of correspondence. doing; rearranging the plurality of particle positions according to the weight; and estimating the position of the moving object by calculating an average value of the rearranged particles.

According to an embodiment, the moving object may be an autonomous vehicle or a vehicle supporting an advanced driver assistance system (ADAS).

An electronic device according to an embodiment includes a communication module configured to receive high definition map data around a moving object and a surrounding image obtained by a photographing device mounted on the moving object; a memory in which computer-executable instructions are stored; and a processor accessing the memory to execute the instructions, wherein the instructions obtain 2D feature point information in the form of a landmark-based probability map from the surrounding image, From the high-precision map data, landmark-based 3D feature point information is obtained, and any one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data is converted into another dimension. and calculating a likelihood between feature point information of the surrounding image and feature point information of the high-precision map data, and estimating the location of the moving object based on the correspondence.

According to an embodiment, the instructions add up probabilities corresponding to each landmark of the feature point information of the high-precision map data in the probability map obtained from the surrounding image, and sum up corresponding to each landmark. It may be configured to calculate the degree of correspondence by multiplying the obtained probabilities.

1A to 1C are diagrams for explaining a method of obtaining feature point information around a moving object according to an exemplary embodiment.
2 is a block diagram of an electronic device according to an exemplary embodiment.
3A and 3B are diagrams for explaining a method of calculating correspondence between feature points of a high-precision map and feature points of a surrounding image according to an exemplary embodiment.
4A and 4B are diagrams for explaining that a degree of correspondence is differently determined according to a candidate position of a moving object according to an exemplary embodiment.
5 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.
6 is a flowchart illustrating a method for calculating a degree of correspondence according to an exemplary embodiment.
7 is a flowchart illustrating a method for estimating a location of a moving object based on a particle filter according to an exemplary embodiment.
8 is a flowchart illustrating a method of estimating a position of a moving object based on a correspondence diagram according to an exemplary embodiment.
9 is a diagram for explaining a system for estimating a position of a moving object according to an exemplary embodiment.
10 is a diagram for explaining a system for estimating a position of a moving object according to an exemplary embodiment.

Specific structural or functional descriptions disclosed in this specification are merely exemplified for the purpose of describing embodiments according to technical concepts, and actual implemented forms may have various other appearances and are limited only to the embodiments described in this specification. It doesn't work.

Terms such as first or second may be used to describe various components, but these terms should only be understood for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may also be termed a first element.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle. Expressions describing the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", etc., should be interpreted similarly.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to indicate that there is an embodied feature, number, step, operation, component, part, or combination thereof, but one or more other features or numbers However, it should be understood that it does not preclude the presence or addition of steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

The embodiments may be implemented in various types of products such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent vehicles, kiosks, and wearable devices. Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference numerals in each figure indicate like elements.

<Electronic device>

1A to 1C are diagrams for explaining a method of obtaining feature point information around a moving object according to an exemplary embodiment.

1A illustrates a video (or image) captured around a moving object according to an example.

Referring to FIG. 1A , various objects serving as references (hereinafter referred to as landmarks) may be included in an image obtained by capturing the front of the moving object. For example, objects corresponding to a yellow lane (or center line) 111, a white lane 113, a curb 115, a street lamp 117, and the like may be included in FIG. 1A . An image as shown in FIG. 1A may be acquired through a photographing device mounted on a mobile body. The photographing device may be a monocular camera, a stereo vision camera, or the like, and may include an image sensor. The image sensor is a semiconductor that converts light entering through a lens of a photographing device into an electrical signal, and may be a color image sensor, a red, green, blue (RGB) sensor, an infrared (IR) sensor, or the like.

1B is a diagram for explaining feature point information extracted from an image around a moving object according to an example.

FIG. 1B may be a landmark-based probability map obtained based on an image around a moving object obtained through a photographing device as shown in FIG. 1A. Referring to FIG. 1B , feature point information 121 , 123 , and 125 classified for each landmark may be obtained from the landmark-based probability map.

For example, the feature point information 121 corresponds to the yellow lane 111 of the image described with reference to FIG. 1A, the feature point information 123 corresponds to the white lane 113, and the feature point information 125 corresponds to the curb 115. can Feature point information of the surrounding image may be obtained according to landmarks based on deep neural network (DNN)-based semantic segmentation.

According to an embodiment, an image around a moving object obtained through a photographing device as shown in FIG. 1A may be input to a learning model for DNN-based semantic segmentation, and feature point information classified according to landmarks may be extracted based on the learning model. there is. The feature point information extracted from the surrounding image obtained by the photographing device may be expressed in the form of a plane, which is a set of pixels on a 2D probability map.

1C is a diagram for explaining feature point information extracted from high-precision map data around a moving object according to an example.

Referring to FIG. 1C , feature point information 131 , 133 , and 135 classified for each landmark may be obtained from high-precision map data around the moving object. For example, the feature point information 131 is a yellow lane around the moving object (eg, the yellow lane 111 in FIG. 1A), and the feature point information 133 is a white lane around the moving object (eg, the yellow lane 111 in FIG. In the white lane 113 , the feature point information 135 may correspond to a curb around the moving object (eg, the curb 115 in FIG. 1A ).

Feature point information obtained from high-precision map data is a three-dimensional coordinate in space and can be classified according to landmarks. For example, referring to FIG. 1C , feature point information 131 includes feature point information corresponding to a yellow lane, feature point information 133 includes feature point information corresponding to a white lane, and feature point information 135 includes a curb. Feature point information corresponding to may be included. Feature point information obtained from high-precision map data may be expressed in a 3-dimensional coordinate form.

According to an embodiment, world domain information on landmarks around the moving object may be transferred from a high-precision map database to an electronic device for estimating the location of the moving object according to location information of the moving object. World domain information may be converted into local domain information for a photographing device.

In order to estimate the position of the moving object, the feature point information (eg, feature point information 121 of FIG. 1B) acquired from the image (eg, FIG. 1A) and the high-precision map data (eg, FIG. 1C) are obtained. A process of comparing one feature point information (eg, feature point information 131 of FIG. 1C) may be required.

According to an embodiment, when an Extended Kalman Filter (EKF) is used for position estimation of a moving object, information about an error between position information predicted based on previous position information and position information measured by a sensor is provided. It may be necessary, and in order to calculate the error, it may be necessary to calculate the degree of similarity (likelihood, hereinafter, correspondence) between the two pieces of information. For example, the feature point information 121 of FIG. 1B may correspond to location information measured by a sensor, and the feature point information 131 of FIG. 1C may correspond to predicted location information.

As described above with reference to FIGS. 1A to 1C, the feature point information obtained from the image obtained by the photographing device, such as the feature point information 121 of FIG. 1B, is 2-dimensional information, and like the feature point information 131 of FIG. Feature point information obtained from high-precision map data may be 3D information. In this case, it may be difficult to calculate the degree of correspondence because the dimensions of the two feature point information do not match.

Calculate the degree of correspondence between the 2D feature point information obtained from the input image described with reference to FIG. 1B and the 3D feature point information obtained from the high-precision map described with reference to FIG. 1C with reference to FIGS. 2 to 10 below. The method is explained in detail.

2 is a block diagram of an electronic device according to an exemplary embodiment.

Referring to FIG. 2 , the electronic device 101 includes a high-precision map database 210 around a moving object, a communication module 190 for communicating with a photographing device 230, a memory 130 storing instructions executable by a computer, and a memory. It may include a processor 120 that accesses 130 and executes instructions.

The electronic device 101 may be mounted on or included in a mobile body to estimate a location, or may be located separately from the mobile body. The electronic device 101 can estimate the position of a moving object, and the moving object may include not only a car, bicycle, etc., but also air vehicles such as drones and airplanes. The mobile body may be an autonomous vehicle or a vehicle supporting an advanced driver assistance system (ADAS).

As described above with reference to FIG. 1B , the photographing device 230 may be a monocular camera, a stereo vision camera, or the like, and may include an image sensor. In FIG. 2 , the photographing device 230 is illustrated for convenience of description, but the photographing device 230 may be an external electronic device or a server in which images of the moving object captured by the photographing device 230 are stored.

The high-precision map database 210 is a database including high definition map (HD map) information, and is transmitted to the electronic device 101 through the communication module 190 according to the request of the electronic device 101. High-precision map information can be transmitted. As described with reference to FIG. 1C, the high-precision map may be composed of 3D feature points classified according to landmarks.

The communication module 190 may support establishment of a direct (eg, wired) communication channel or wireless communication channel between the electronic device 101 and an external electronic device (or server), and communication through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 190 is a wireless communication module (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (eg, a local area network (LAN)). ) communication module, or power line communication module). Among these communication modules, the corresponding communication module is a short-distance communication network (eg Bluetooth, wireless fidelity (WiFi) direct or IrDA (infrared data association)) or a long-distance communication network (eg legacy cellular network, 5G network, next-generation communication network, Internet , or through a computer network (such as a LAN or WAN) to communicate with an external electronic device. These various types of communication modules may be integrated as one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips).

The wireless communication module may support a 5G network after a 4G network and a next-generation communication technology, for example, NR access technology (new radio access technology). NR access technologies include high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and access of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low latency (URLLC)). -latency communications)) can be supported. The wireless communication module may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module uses various technologies for securing performance in a high frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), and full-dimensional multiple-output (FD). Technologies such as full dimensional MIMO (MIMO), array antenna, analog beam-forming, or large scale antenna may be supported. The wireless communication module may support various requirements defined for the electronic device 101, an external electronic device, or a network system. According to one embodiment, the wireless communication module may be used to realize peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency (eg, down) for realizing URLLC. link (DL) and uplink (UL) each of 0.5 ms or less, or round trip 1 ms or less) may be supported.

A method of the communication module 190 communicating with the photographing device 230 to obtain 2D feature point information and a method of communicating with the high precision map database 210 to obtain 3D feature point information may be the same or different.

According to an embodiment, the processor 120 may obtain 2D feature point information in the form of a landmark-based probability map from a surrounding image acquired by the photographing device 230 mounted on a moving object. As described above with reference to FIG. 1B , the processor 120 receives an image around the moving object from the photographing device 230 or an external electronic device or server in which the image is stored, and based on DNN-based semantic segmentation, the 2D feature point for each landmark. information can be obtained.

According to another embodiment, a process of extracting 2D feature point information from an image around a moving object is performed in an external electronic device or server, and the processor 120 receives the extracted 2D feature point information in the form of a probability map based on landmarks. can

According to an embodiment, the processor 120 may obtain landmark-based 3D feature point information of a high-precision map around the moving object. As described above with reference to FIG. 1C , the processor 120 may obtain 3D feature point information of a high-precision map around the moving object by requesting the 3D feature point information from the high-precision map database 210 based on the location information of the moving object. .

According to an embodiment, feature point information stored in the high-precision map database 210 may be coordinates on a world domain, and the processor 120 determines landmarks around the moving object received from the high-precision map database 210. The feature point information for the image may be converted into a local domain for the photographing device. The conversion process from the world domain to the local domain will be described in detail with reference to [Equation 1] below.

According to another embodiment, conversion from the world domain to the local domain is performed in an external electronic device or server, and 3D feature point information in the form of a local domain may be transmitted to the processor 120 .

In order to calculate the degree of correspondence between 2D feature point information and 3D feature point information, the processor 120 converts any one dimension of the 2D feature point information of surrounding images and 3D feature point information of high-precision map data into another dimension. can do.

For example, the processor 120 may project 3D feature point information of high-precision map data into a 2D probability map obtained from a surrounding image based on perspective mapping. The 3D feature point information of the high-precision map data may be projected into a 2D probability map obtained from a surrounding image based on [Equation 1] and [Equation 2] below.

According to an embodiment, based on [Equation 1], 3D feature point information of the high-precision map data may be converted from the world domain to the local domain based on the photographing device. This may correspond to the conversion operation from the world domain to the local domain described above in relation to the high-precision map database 210 .

은 고정밀 지도 데이터의 월드 도메인 상 3차원 특징점 정보의 좌표일 수 있다. 예를 들어, 도 1c의 랜드마크 별 특징점들(131, 133, 135) 중 하나의 좌표일 수 있다.

는 고정밀 지도 데이터의 월드 도메인 상 이동체의 3차원 좌표일 수 있다. 예를 들어, 도 1c의 차량의 월드 도메인 상 3차원 좌표일 수 있다.

는 이동체의 월드 도메인 대비 회전변환행렬일 수 있다.

는 촬영 장치(230)의 이동체 대비 회전변환행렬일 수 있다.

는 촬영 장치(230)의 이동체 대비 상대 거리일 수 있다. [수학식 1]에 기초하여, 촬영 장치(230)를 기준으로 한 로컬 도메인 상의 3차원 특징점 정보의 각 성분인

가 획득될 수 있다.

may be coordinates of 3D feature point information on the world domain of high-precision map data. For example, it may be the coordinates of one of feature points 131, 133, and 135 for each landmark of FIG. 1C.

may be the 3D coordinates of a moving object on the world domain of high-precision map data. For example, it may be a 3D coordinate on the world domain of the vehicle of FIG. 1C.

may be a rotation transformation matrix relative to the world domain of the moving object.

may be a rotation transformation matrix relative to the moving object of the photographing device 230 .

may be a relative distance of the photographing device 230 to the moving object. Based on [Equation 1], each component of the 3D feature point information on the local domain based on the photographing device 230

can be obtained.

According to an embodiment, based on [Equation 2], 3D feature point information converted to a photographing device reference local domain may be projected onto a 2D probability map.

가 2차원 확률 맵으로 투영되고, 2차원 좌표

가 획득될 수 있다.

는 각각 촬영 장치의 가로 방향 초점 거리, 세로 방향 초점 거리일 수 있다.Based on the above [Equation 2], the 3D feature point information obtained in [Equation 1] above

is projected into a two-dimensional probability map, and the two-dimensional coordinates

can be obtained.

may be a focal length in a horizontal direction and a focal length in a vertical direction of the photographing device, respectively.

As another example, the processor 120 may convert 2D feature point information of the surrounding image into a 3D probability map form based on inverse perspective mapping. Since depth information is required to convert a 2D image into a 3D image, the photographing device 230 may be a stereo vision camera.

According to an embodiment, the processor 120 may match the dimensions of the two feature point information, calculate a degree of correspondence between the feature point information, and estimate the position of the moving object based on the degree of correspondence. The processor 120 may calculate a degree of correspondence by summing probabilities corresponding to each landmark of the feature point information of the high-precision map data and multiplying the summed probabilities in the probability map obtained from the image surrounding the moving object.

As an example, as described above, when the dimensions are matched in 2D, when 3D feature point information is projected onto a 2D probability map, probabilities corresponding to the projected 2D coordinates are summed for each landmark, and the summed probabilities are multiplied to respond. figure can be calculated. As another example, in the case of matching dimensions in 3D, a 2D probability map may be converted into a 3D probability map according to an inverse perspective transformation method. Probabilities corresponding to the 3D feature point information on the converted 3D probability map may be summed for each landmark, and the degree of correspondence may be calculated by multiplying the summed probabilities.

Based on the calculated degree of correspondence, the processor 120 may estimate the position of the moving object or update the estimation result according to a particle filter or maximum likelihood (ML) optimization method.

An operation of calculating the degree of correspondence of the processor 120 and an operation of estimating the position of a moving object based on the degree of correspondence according to various embodiments will be described in detail with reference to FIGS. 5 to 9 .

3A and 3B are diagrams for explaining a method of calculating correspondence between feature points of a high-precision map and feature points of a surrounding image according to an exemplary embodiment.

As described above with reference to FIG. 2 , the processor 120 of the electronic device 101 matches dimensions between two feature points, sums corresponding probabilities for each landmark to calculate a degree of correspondence, and adds the summed probabilities. The correspondence can be calculated by multiplying

FIG. 3A is a diagram for explaining a method of determining a degree of correspondence by summing probabilities without classifying each landmark, and FIG. 3B is a diagram for explaining a method of determining a degree of correspondence by multiplying probabilities classified by landmarks. it is a drawing

As described above with reference to FIG. 2, it is possible to match dimensions in three dimensions, but for convenience of description, a method of calculating a degree of correspondence in dimensions matched in two dimensions is described in FIGS. 3A and 3B.

Referring to FIG. 3A , a situation 310 in which 3D feature point information is projected onto a landmark-based 2D probability map according to an embodiment is illustrated.

As described above with reference to FIG. 2, [Equation 1] and [Equation 2], 3D feature points may be projected onto a 2D probability map based on the perspective transformation method. 310 of FIG. 3 may be obtained by projecting 3D feature point information onto a 2D probability map obtained from an image around a moving object acquired by a photographing device, like the 2D probability map described with reference to FIG. 1B .

Referring to FIG. 3A , 2D feature point information on a 2D probability map and projected 3D feature point information are shown. For example, the 2D feature point information 311 may correspond to a lane of the moving object front image, and the 2D feature point information 313 may correspond to a stop line of the moving object front image. As a result of projecting the 3D feature points, in the case of lanes, the 3D feature point information 321 projected to the 2D feature point information 311 corresponds a lot, but in the case of a stop line, the 2D feature point information 313 and the projected 3D feature points It can be seen that information 323 does not correspond at all.

If the degree of correspondence is calculated by simply summing the probabilities corresponding to the projected 3D feature points instead of separately calculating the degree of correspondence for each landmark, accuracy of the degree of correspondence may decrease. For example, even though stop lines do not correspond at all, since lanes having a relatively large number of feature points correspond well, the overall degree of correspondence can be calculated relatively high.

According to an embodiment, the processor 120 of the electronic device 101 calculates the degree of correspondence by summing the probabilities for each landmark and multiplying the summed probabilities, thereby improving the accuracy of the degree of correspondence. A process of calculating a degree of correspondence by dividing according to landmarks, like the processor 120 of the electronic device 101, will be described in detail with reference to FIG. 3B.

Referring to FIG. 3B , situations 340 and 360 in which 3D feature point information is projected into a landmark-based 2D probability map according to an embodiment are separately shown for each landmark.

340 and 360 in FIG. 3B are only for explaining the process of calculating the degree of correspondence by classifying each landmark, and since both FIGS. 3A and 3B differ only in the method of calculating the degree of correspondence in the same situation, the processor in the actual calculation process ( 120) may calculate a degree of correspondence based on one probability map as shown in 310 of FIG. 3A.

Referring to 340 of FIG. 3B , 2D feature point information for a stop line among landmarks such as lanes and stop lines on a 2D probability map and projected 3D feature point information for the stop line are shown. For example, the 2D feature point information 343 may correspond to a stop line in the 2D probability map obtained from the front image of the moving object, and the 3D feature point information 353 corresponding to the stop line projected on the 2D probability map is shown. there is. Referring to 340 of FIG. 3B , it can be seen that the 3D feature point information 353 projected onto the 2D probability map does not correspond to the 2D feature point information 343 at all.

Referring to 360 of FIG. 3B , 2D feature point information for lanes and projected 3D feature point information for lanes are shown on the 2D probability map. For example, the 2D feature point information 361 may correspond to a lane in the 2D probability map obtained from the front image of the moving object, and the 3D feature point information 371 corresponding to the lane projected on the 2D probability map is shown. there is. Referring to 360 of FIG. 3B , it can be seen that the 3D feature point information 371 projected on the 2D probability map corresponds well to the 2D feature point information 361 .

The processor 120 of the electronic device 101 may calculate a degree of correspondence by summing probabilities corresponding to each projected 3D feature point for each landmark on the probability map and multiplying the summed probabilities for each landmark. Referring to 340 in FIG. 3B , since all three feature points projected on the probability map do not correspond to the 2D feature point information 343 for the landmark stop line, the processor 120 sums the probabilities corresponding to the three feature points. can be close to 0. Referring to 360 of FIG. 3B , a high value may be derived by summing probabilities for landmark lanes.

Unlike the description with reference to FIG. 3A , the processor 120 may calculate a degree of correspondence by multiplying probabilities obtained by classifying and summing landmarks for each landmark as shown in FIG. 3B . Through this, even if the probability calculated for the lane is high, if the probability calculated for the stop line is close to 0, the degree of correspondence calculated by multiplying the two values may be smaller than the degree of correspondence calculated in FIG. 3A.

According to an embodiment, the processor 120 may calculate a more accurate degree of correspondence and increase the uniqueness of the degree of correspondence by classifying each landmark and calculating the degree of correspondence. A specific process of calculating the degree of correspondence will be described in detail with reference to FIG. 6 .

4A and 4B are diagrams for explaining that a degree of correspondence is differently determined according to a candidate position of a moving object according to an exemplary embodiment.

Referring to FIGS. 4A and 4B , examples in which 3D information of high-precision map data as shown in FIG. 1C are projected onto a 2D probability map as shown in FIG. 1B are shown.

As described above with reference to FIG. 2 , it is possible to match dimensions in three dimensions, but for convenience of description, an embodiment in which two dimensions are matched will be described in FIGS. 4A and 4B .

Referring to FIG. 4A , 2D feature point information 411 and projected 3D feature point information 431 corresponding to a yellow lane (or center line), 2D feature point information 413 and 3D feature point corresponding to a white lane Information 433, and 2D feature point information 415 and 3D feature point information 435 corresponding to the curb are shown. In this case, since the degree of correspondence is low, when the degree of correspondence is calculated as described above with reference to FIGS. 2 to 3B , the degree of correspondence may be calculated relatively low.

Referring to FIG. 4B, 2D feature point information 461 and projected 3D feature point information 481 corresponding to the yellow lane (or center line), 2D feature point information 463 and 3D feature point corresponding to the white lane Information 483, and 2D feature point information 465 and 3D feature point information 485 corresponding to the curb are shown. In this case, since the degree of correspondence is higher than that of FIG. 4A, when the degree of correspondence is calculated as described above with reference to FIGS. 2 to 3B, the degree of correspondence may be calculated relatively high.

The processor 120 may estimate the location of the moving object based on the degree of correspondence. According to an embodiment, the processor 120 may set a higher weight for a candidate location corresponding to FIG. 4B having a relatively high correspondence than a candidate location corresponding to FIG. 4A having a relatively low correspondence. The processor 120 may update a result of estimating the position of the moving object according to a particle filter or a maximum likelihood (ML) optimization method based on the degree of correspondence. A method of estimating the location of a moving object by the processor 120 based on the particle filter will be described in detail with reference to FIGS. 7 and 8 .

<Operation method of electronic device>

5 is a flowchart illustrating a method of operating an electronic device according to an exemplary embodiment.

Operations 510 to 550 may be performed by the processor 120 of the electronic device 101 described above with reference to FIG. 2 , and for concise description, contents overlapping those described with reference to FIGS. 1 to 4B are may be omitted.

According to an embodiment, in operation 510, the processor 120 may obtain 2D feature point information in the form of a probability map from an image surrounding the moving object. As described above with reference to FIG. 2 , the image around the moving object may be acquired by the photographing device 230, and the 2D feature point information in the form of a probability map may be obtained based on DNN-based semantic segmentation.

According to an embodiment, in operation 520, the processor 120 may obtain 3D feature point information from high-precision map data around the moving object. As described above with reference to FIG. 2 , the processor 120 may request high-precision map data around the moving object from the high-precision map database 210 based on the moving object location information, and obtain 3D feature point information of the high-precision map data. there is. The processor 120 may convert the 3D feature point information expressed in the world domain into the local domain based on [Equation 1] described above.

According to an embodiment, in operation 530, the processor 120 may convert one dimension of 2D feature point information and 3D feature point information into another dimension. As described above with reference to FIG. 2, the processor 120 projects 3D feature points into a 2D probability map based on the perspective transformation method and transforms them into 2D, or transforms the 2D probability map into 3D based on the inverse perspective transformation method. It can be converted into a dimensional probability map form. For example, the processor 120 may project the 3D feature points into a 2D probability map based on Equation 2 described above.

According to an embodiment, in operation 540, the processor 120 may calculate a degree of correspondence between feature point information of the surrounding image and feature point information of the high-precision map data. As described above with reference to FIGS. 3A and 3B , when 3D feature points are projected onto a 2D probability map, the processor 120 sums probabilities corresponding to the projected coordinates for each landmark, and multiplies the summed probabilities to respond. figure can be calculated. When the 2D probability map is converted into a 3D probability map, the processor 120 may calculate a degree of correspondence by summing probabilities corresponding to coordinates of the 3D feature point information for each landmark and multiplying the summed probabilities. An operation of calculating the degree of correspondence by the processor 120 will be described in detail with reference to FIG. 6 .

According to an embodiment, in operation 550, the processor 120 estimates the position of the moving object based on the degree of correspondence. According to an embodiment, as described with reference to FIGS. 4A and 4B , the processor 120 may estimate the position of the moving object by setting different weights for the predicted position according to the degree of correspondence.

6 is a flowchart illustrating a method for calculating a degree of correspondence according to an exemplary embodiment.

Operations 610 to 620 may be performed by the processor 120 of the electronic device 101 described above with reference to FIG. 2 , and for concise description, contents overlapping those described with reference to FIGS. 1 to 5 are may be omitted.

According to an embodiment, operations 610 to 620 may correspond to an operation of calculating a degree of correspondence between feature point information of the surrounding image and feature point information of the high-precision map data described with reference to FIG. 5 (eg, operation 540 of FIG. 5 ). can

According to an embodiment, in operation 610, the processor 120 may add probabilities corresponding to respective landmarks of feature point information of the high-precision map data in the probability map obtained from the surrounding images. As described with reference to FIG. 3B , the processor 120 may add probabilities corresponding to feature point information of high-precision map data on a probability map obtained from surrounding images for each landmark.

According to an embodiment, in operation 620, the processor 120 may calculate a degree of correspondence by multiplying the summed probability corresponding to each landmark. As described with reference to FIG. 3B , the processor 120 may obtain a more accurate degree of correspondence by multiplying the summed probabilities for each landmark to calculate the degree of correspondence.

Operations 610 and 620 of the processor 120 may be expressed as [Equation 3] below. [Equation 3] below may be a method of calculating correspondence when 3D feature point information of a high-precision map is projected onto a 2D probability map based on [Equation 1] and [Equation 2] described above.

는 전술한 [수학식 2]에서 2차원 확률 맵으로 투영된 고정밀 지도 데이터의 3차원 특징점 정보일 수 있다. 2차원 확률 맵의 모든 좌표에는 대응하는 확률이 존재하고,

는 2차원 확률 맵의 좌표에 대응하는 확률일 수 있다. 일례로,

는 Gaussian Blur filter일 수 있다.

은 특징점에 대한 인덱스(index)일 수 있다.

는 랜드마크에 대한 인덱스일 수 있다.

may be 3D feature point information of the high-precision map data projected onto the 2D probability map in [Equation 2] above. All coordinates of the two-dimensional probability map have corresponding probabilities,

may be a probability corresponding to the coordinates of a 2-dimensional probability map. For example,

may be a Gaussian Blur filter.

may be an index for a feature point.

may be an index for a landmark.

에 대응하는 모든 특징점들, [수학식 3] 상에서는 인덱스 넘버 m부터 M 까지의 특징점들에 대응하는 확률들을 합산(∑)할 수 있다. 전술한 도 3b를 참조하면, 부재번호 340 및 부재번호 360 각각이 랜드마크 하나에 대한 대응하는 특징점들이 도시된 예일 수 있다. Referring to [Equation 3], in operation 610, the processor 120 determines the landmark

Probabilities corresponding to all feature points corresponding to , and feature points from index numbers m to M in [Equation 3] can be summed (∑). Referring to FIG. 3B described above, each of reference number 340 and reference number 360 may be an example in which feature points corresponding to one landmark are shown.

Referring back to [Equation 3], in operation 620, the processor 120 may multiply (∏) the summed probabilities from landmark i to I. As described above with reference to FIG. 3B, the uniqueness of the degree of correspondence can be increased by dividing the degree of correspondence according to each landmark and calculating the degree of correspondence.

According to an embodiment, the processor 120 may calculate a degree of correspondence (or likelihood) between feature points of different dimensions by performing operations 610 to 620.

7 is a flowchart illustrating a method for estimating a location of a moving object based on a particle filter according to an exemplary embodiment.

Operations 710 to 760 may be performed by the processor 120 of the electronic device 101 described above with reference to FIG. 2 , and contents overlapping those described with reference to FIGS. 1 to 6 for concise description are may be omitted.

Operations 710 to 760 may correspond to a method in which the processor 120 of the electronic device 101 estimates the position of the moving object based on the particle filter. Operations 710 to 760 may be specific operations based on a particle filter in relation to the operations described above with reference to FIG. 5 .

A 'particle' is software-wisely placed on a digital indoor map, and has one or more properties. Each particle preferably has at least two properties, 'direction' and 'position'. Each particle is moved by reflecting the relative movement of the moving object, and is deleted if it is in a position where it cannot be moved on the digital indoor map after the movement. In addition, as a result of particle movement, when the total number of particles falls below the reference value, new particles having the same properties as the existing particles except for their positions are regenerated. The method for estimating a position based on particles may largely include a process of arranging particles at candidate positions, a process of calculating a likelihood for each particle, and a process of resampling the distribution of particles based on the calculated likelihood. can Since the location measurement method using 'particle filtering' is widely known, detailed operation of the particle filter will be omitted.

Operations 710 and 720 respectively correspond to operations 510 and 520 described with reference to FIG. 5 , and since they overlap with those described with reference to FIG. 5 , descriptions thereof are omitted.

According to an embodiment, in operation 730, the processor 120 may predict positions of a plurality of particles corresponding to candidate positions of the moving object. According to an embodiment, the processor 120 may predict a plurality of particle positions corresponding to candidate positions of the current moving object based on particle position information relocated from a previous time point and movement displacement of the moving object from the previous time point to the present. . The movement displacement of the moving object from the previous point in time to the present may be obtained based on, for example, wheel speed information.

According to an embodiment, in operation 740, the processor 120 may project 3D feature point information onto a 2D probability map for each of a plurality of particle positions. That is, there may be one 2D probability map corresponding to each predicted particle position and a 3D feature point of a high-precision map projected on the 2D probability map. As an example, FIG. 4A is a 2D probability map and projected 3D feature points corresponding to an arbitrary particle position, and FIG. 4B may be a 2D probability map and projected 3D feature points corresponding to another arbitrary particle position. . The method for the processor 120 to project the 3D feature point information onto the 2D probability map in operation 740 is omitted because it overlaps with those described with reference to FIG. 2, [Equation 1], [Equation 2], and operation 530 of FIG. do.

According to an embodiment, in operation 750, the processor 120 may calculate a degree of correspondence between 3D feature point information projected on the probability map and 2D feature point information of the probability map, for each of a plurality of particle positions. Operation 750 is omitted because it overlaps with operations 760 of FIGS. 3A, 3B, and 5 and those described with reference to [Equation 3].

According to an embodiment, in operation 760, the processor 120 may estimate the position of the moving object by rearranging the plurality of particle positions based on the degree of correspondence. A specific operation of estimating the position of a moving object based on the particle filter will be described in detail with reference to FIG. 8 .

8 is a flowchart illustrating a method of estimating a position of a moving object based on a correspondence diagram according to an exemplary embodiment.

Operations 810 to 830 may be performed by the processor 120 of the electronic device 101 described above with reference to FIG. 2 , and contents overlapping those described with reference to FIGS. 1 to 7 for concise description are may be omitted.

According to an embodiment, operations 810 to 840 may correspond to an operation of estimating a position of a moving object based on the particle filter described with reference to FIG. 7 (eg, operation 760 of FIG. 7 ).

According to an embodiment, in operation 810, the processor 120 may set a weight for each particle position according to the degree of correspondence. For example, for two particle positions corresponding to two candidate positions of the moving object, a 2D probability map and a projected 3D feature point may be obtained as shown in FIGS. 4A and 4B, respectively. The processor 120 may set a low weight for the corresponding particle in the case of FIG. 4A having a low degree of correspondence, and set a high weight for the corresponding particle in the case of FIG. 4B having a high degree of correspondence.

According to an embodiment, in operation 820, the processor 120 may resample a plurality of particle positions according to weights. Since a high weight is set for a particle position corresponding to FIG. 4B among a plurality of predicted particle positions, more particles may be disposed at a position corresponding to FIG. 4B.

According to an embodiment, in operation 830, the processor 120 may estimate the position of the moving object by calculating an average value of the rearranged particles. For example, the processor 120 may estimate the position of the moving object based on maximum a posterior (MAP).

9 is a diagram for explaining a system for estimating a position of a moving object according to an exemplary embodiment.

Referring to FIG. 9, a moving object position estimation system 900 composed of a fusion part 902 based on an extended Kalman filter 990 and a map matching part 901 based on a particle filter 980. ) is shown.

There is a method of utilizing the convergence of GPS and INS (Inertial Navigation System, 930) as a location estimation method for estimating the location of an autonomous vehicle in the world domain. Accuracy cannot be guaranteed in complex urban environments. In order to eliminate GPS dependence, the mobile position estimation system 900 proposes a new position estimation method by converging an Inertial Measurement Unit (IMU) 920, a wheel odometry 910, and a high-precision map 940.

An Extended Kalman Filter (EKF) 990 of the synthesis unit 902 is a filter generally used to fuse different sensors in a non-linear relationship, and the use of the Extended Kalman Filter 990 For this, a system model and a measurement model must be defined. In the moving object position estimation system 900, a system model and a measurement model may be defined based on the wheel odometry 910, the IMU 920, and position information. The system model predicts a state variable and covariance based on the mathematical model, and the predicted state variable and covariance are compensated by the difference between the measured value and the predicted measured value.

However, in the case of a probability map, it may be ambiguous to calculate the difference between the measured value and the predicted measured value. As described above with reference to FIGS. 1 to 8 , the processor 120 of the electronic device 101 calculates the likelihood between the predicted value and the measured value, thereby determining the relationship between the measured value and the predicted measured value in the probability map. difference can be calculated.

As described above with reference to FIGS. 1A to 1C , when an Extended Kalman Filter (EKF) is used for position estimation of a moving object, position information predicted based on previous position information and position information measured by a sensor Information on an error between the two information may be required, and a similarity (likelihood, hereinafter referred to as correspondence) between two pieces of information may need to be calculated in order to calculate the error. For example, the feature point information 121 of FIG. 1B may correspond to location information measured by a sensor, and the feature point information 131 of FIG. 1C may correspond to predicted location information.

The operation of the electronic device 101 described with reference to FIGS. 1 to 8 may correspond to the operation of the map matching unit 901 in the moving object position estimation system 900 . In the map matching unit 901, an operation in which the DNN-based semantic segmentation 970 is performed on the surrounding image acquired from the camera 960 is based on landmarks from the surrounding image acquired by the processor 120 from the photographing device 230. It may correspond to an operation of obtaining 2D feature point information in the form of a probability map (eg, operation 510 of FIG. 5 and operation 710 of FIG. 7 ). In the operation 950 of extracting landmarks around the moving object from the high-precision map 940 in the map matching unit 901, the processor 120 receives high-precision map data around the moving object from the high-precision map database 210, and local It may correspond to an operation of converting to a domain (eg, operation 520 of FIG. 5 and operation 720 of FIG. 7 ). In the map matching unit 901, the operation of the particle filter 980 is an operation in which the processor 120 calculates the degree of correspondence between the feature points of the high-precision map and the feature points of the probability map based on the particles and estimates the position of the moving object (eg : This may correspond to operations 730 to 760 of FIG. 7 ).

10 illustrates a vehicle equipped with an electronic device according to an exemplary embodiment.

Referring to FIG. 10 , a vehicle 1000 including the electronic device 101 described with reference to FIGS. 1 to 9 is shown. The vehicle 1000 according to an embodiment may drive in an autonomous mode according to a recognized driving environment even when there is little or no input from a driver. The driving environment may be recognized through one or more sensors attached or installed in the vehicle 1000 . For example, one or more sensors may include, but are not limited to, camera, LIDAR, RADAR and voice recognition sensors. The driving environment may include roads, road conditions, lane types, presence or absence of surrounding vehicles, distance from adjacent vehicles, weather, presence or absence of obstacles, and the like, but is not limited to the described examples.

The vehicle 1000 recognizes the driving environment and creates an autonomous driving route suitable for the driving environment. Self-driving vehicles control internal and external mechanical elements to follow the self-driving path. The vehicle 1000 may periodically generate an autonomous driving route.

According to another aspect, the vehicle 1000 may assist a driver's driving by using an advanced driver assistance system (ADAS). ADAS includes the Autonomous Emergency Braking (AEB) system, which reduces or stops the speed on its own without the driver stepping on the brakes when there is a risk of collision, and the Lane Keep Assist system, which maintains the lane by adjusting the driving direction when the driver leaves the lane. System: LKAS), Advanced Smart Cruise Control (ASCC), which automatically maintains a distance from the vehicle in front while running at a pre-set speed, Blind-Spot Collision Avoidance Assistance that detects the risk of collision in a blind spot and helps to change lanes safely system (Active Blind Spot Detection: ABSD), and the Around View Monitoring System (AVM) that visually displays the situation around the vehicle.

The electronic device 101 included in the vehicle 1000 controls mechanical devices of the vehicle 1000 to autonomously drive or assist a driver's driving, and may include ECUs other than the described embodiments, various types of controllers or sensors, etc. can be used

According to an embodiment, the electronic device 101 described with reference to FIGS. 1 to 9 may be included in an autonomous vehicle or a vehicle 1000 supporting an advanced driver assistance system, and the electronic device 101 may be included in the vehicle 1000 ) position estimation can be performed.

In the above-described embodiments, the memory 130 may store various pieces of information generated in the process of the processor 120 described above. In addition, the memory 130 may store various data and programs. The memory 130 may include volatile memory or non-volatile memory. The memory 720 may include a mass storage medium such as a hard disk to store various types of data.

The processor 120 may be a hardware-implemented device having a circuit having a physical structure for executing desired operations. For example, desired operations may include codes or instructions included in a program. For example, classification units implemented in hardware include a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a processor core, and a multi-core processor. (multi-core processor), multiprocessor, ASIC (Application-Specific Integrated Circuit), FPGA (Field Programmable Gate Array), NPU (Neural Processing Unit), and the like.

The processor 120 may execute a program and control an electronic device. Program codes executed by the processor may be stored in memory.

The embodiments described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, the devices, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic units (PLUs), microprocessors, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in the art of computer software. may be Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or the components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (20)

In the method of operating an electronic device,
obtaining 2D feature point information in the form of a probability map based on landmarks from a surrounding image acquired by a photographing device mounted on a moving object;
obtaining landmark-based 3D feature point information from high definition map data around the moving object;
converting one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data into another dimension;
calculating a likelihood between feature point information of the surrounding image and feature point information of the high-precision map data; and
Estimating the position of the moving object based on the degree of correspondence
including,
Methods of operating electronic devices.
According to claim 1,
The 2D feature point information of the surrounding image is obtained according to a landmark based on deep neural network (DNN)-based semantic segmentation.
Methods of operating electronic devices.
According to claim 1,
Obtaining the 3D feature point information from high definition map data around the moving object,
receiving 3D feature point information on a world domain for landmarks around the moving object from a high-precision map database based on location information of the moving object; and
converting 3D feature point information on the world domain into a local domain for the photographing device;
including,
Methods of operating electronic devices.
According to claim 1,
The step of converting one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data into another dimension,
Converting 2D feature point information of the surrounding image into a 3D probability map form based on inverse perspective mapping
including,
Methods of operating electronic devices.
According to claim 1,
The step of converting one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data into another dimension,
Projecting the 3D feature point information of the high-precision map data into a 2D probability map obtained from the surrounding image based on perspective mapping
including,
Methods of operating electronic devices.
According to claim 1,
The step of calculating the likelihood between the feature point information of the surrounding image and the feature point information of the high-precision map data,
summing probabilities corresponding to each landmark of feature point information of the high-precision map data in a probability map acquired from the surrounding image; and
Calculating the degree of correspondence by multiplying the summed probabilities corresponding to each landmark
including,
Methods of operating electronic devices.
According to claim 1,
The step of estimating the position of the moving object based on the degree of correspondence,
Updating a position estimation result of the moving object according to a particle filter or maximum likelihood (ML) optimization method based on the degree of correspondence
including,
Methods of operating electronic devices.
According to claim 1,
The mobile body is an autonomous vehicle or a vehicle supporting an advanced driver assistance system (ADAS),
Methods of operating electronic devices.
According to claim 1,
The landmark includes at least one of a white lane, a yellow lane, a crosswalk, a speed bump, a traffic light, and a traffic sign.
Methods of operating electronic devices.
In the method for estimating the position of a moving object based on a particle filter,
obtaining 2D feature point information in the form of a probability map based on landmarks from a surrounding image acquired by a photographing device mounted on a moving object;
obtaining landmark-based 3D feature point information from high definition map data around the moving object;
predicting a plurality of particle positions corresponding to candidate positions of the moving object;
Projecting the 3D feature point information onto a probability map obtained from the surrounding image for each of the plurality of particle positions;
calculating a likelihood between the 3D feature point information projected on the probability map and the 2D feature point information of the probability map, for each of the plurality of particle positions; and
Estimating the position of the moving object by rearranging the positions of the plurality of particles based on the degree of correspondence.
including,
A method for estimating the position of a moving object.
According to claim 10,
The two-dimensional feature point information is obtained according to a landmark based on semantic segmentation based on a deep neural network (DNN).
A method for estimating the position of a moving object.
According to claim 10,
Obtaining the 3D feature point information from high definition map data around the moving object,
receiving 3D feature point information on a world domain for landmarks around the moving object from a high-precision map database based on location information of the moving object; and
converting 3D feature point information on the world domain into a local domain for the photographing device;
including,
A method for estimating the position of a moving object.
According to claim 10,
The step of predicting the plurality of particle positions,
Predicting the plurality of particle positions based on particle position information relocated from a previous viewpoint and movement displacement of the moving object from the previous viewpoint.
including,
A method for estimating the position of a moving object.
According to claim 10,
The 3D feature point information is projected into a 2D probability map obtained from the surrounding image based on perspective mapping.
A method for estimating the position of a moving object.
According to claim 10,
Calculating a likelihood between the 3D feature point information projected on the probability map and the 2D feature point information of the probability map,
summing probabilities corresponding to each landmark of the projected 3D feature point information in the probability map; and
Multiplying the summed probabilities corresponding to each landmark
including,
A method for estimating the position of a moving object.
According to claim 1,
The step of estimating the position of the moving object by rearranging the positions of the plurality of particles based on the degree of correspondence,
setting a weight for each of the plurality of particle positions according to the degree of correspondence;
rearranging the plurality of particle positions according to the weight; and
Estimating the position of the moving object by calculating an average value of the rearranged particles
including,
A method for estimating the position of a moving object.
According to claim 10,
The mobile body is an autonomous vehicle or a vehicle supporting an advanced driver assistance system (ADAS),
A method for estimating the position of a moving object.
A computer program stored in a medium to execute the method of any one of claims 1 to 17 in combination with hardware.
In electronic devices,
A communication module for receiving high-definition map data around the moving object and surrounding images acquired by a photographing device mounted on the moving object;
a memory in which computer-executable instructions are stored; and
A processor that accesses the memory and executes the instructions
including,
These commands are
Obtaining 2D feature point information in the form of a landmark-based probability map from the surrounding image;
Obtaining landmark-based 3D feature point information from the high-precision map data,
converting one dimension of the 2D feature point information of the surrounding image and the 3D feature point information of the high-precision map data into another dimension;
Calculate a likelihood between feature point information of the surrounding image and feature point information of the high-precision map data;
Estimating the position of the moving object based on the degree of correspondence
configured to
electronic device.
According to claim 19,
These commands are
Summing probabilities corresponding to each landmark of the feature point information of the high-precision map data in the probability map obtained from the surrounding image;
Calculate the degree of correspondence by multiplying the summed probabilities corresponding to each landmark
configured to
electronic device.

Images