KR20210099752A - Electronic device for conforming pose of street view image based on two-dimension map information and operating method thereof - Google Patents

Abstract

According to various embodiments, provided are an electronic device and an operating method thereof. The present invention is to match a pose of a street view image based on two-dimensional map information. The operating method is configured to acquire two-dimensional map information for the outdoor environment and a plurality of street view images, use the street view images to generate a three-dimensional map, use the two-dimensional map information to match the three-dimensional map, and provide a visual localization of the outdoor environment based on the matched three-dimensional map information.

Description

An electronic device for matching poses of a street view image based on two-dimensional map information and an operation method thereof

Various embodiments relate to an electronic device for matching poses of a street view image based on 2D map information and an operating method thereof.

In general, visual localization (VL) has been proposed to provide navigation in an indoor environment. For example, the robot may perform visual localization using map information for an indoor environment, and may move indoors through this. Currently, research is being conducted to extend visual localization to outdoor environments. For visual localization in an outdoor environment, 3D map information for the outdoor environment is required. However, there is a problem in that various errors occur in providing visual localization based on 3D map information for an outdoor environment. For example, when cursoring is performed on 3D map information using 2D map information, the shape or size of an item related to the cursoring may be distorted and displayed.

Various embodiments provide an electronic device for providing visual localization in an outdoor environment based on 3D map information on the outdoor environment, and an operating method thereof.

Various embodiments provide an electronic device capable of processing 3D map information based on 2D map information while generating 3D map information, and an operating method thereof.

Various embodiments provide an electronic device for matching a pose of a street view image based on 2D map information, and an operating method thereof.

A method of operating an electronic device according to various embodiments includes an operation of acquiring 2D map information for an outdoor environment and a plurality of street view images, and 3D map information by using the 2D map information and the street view images. and providing visual localization for the outdoor environment based on the 3D map information.

An electronic device according to various embodiments includes a memory and a processor connected to the memory and configured to execute at least one instruction stored in the memory, wherein the processor includes two-dimensional map information for an outdoor environment and a plurality of obtains street view images, generates 3D map information using the 2D map information and the street view images, and provides visual localization for the outdoor environment based on the 3D map information can be configured to

According to various embodiments, the electronic device may provide visual localization in the outdoor environment by generating 3D map information on the outdoor environment. That is, the electronic device may perform positioning in the outdoor environment based on 3D map information on the outdoor environment. In this case, by matching the poses of the street view image required for the electronic device to generate the 3D map information based on the 2D map information, an error occurring when providing the visual localization may be minimized. For example, when cursoring is performed on 3D map information using 2D map information, distortion of a shape or size of an item related to the cursoring may be minimized. In addition, in providing augmented reality (AR) navigation based on visual localization, a point of interest (POI) of two-dimensional map information (eg, a store, a crosswalk, a building) etc.) data can be used.

1 is a diagram illustrating an electronic device according to various embodiments of the present disclosure;
2 is a diagram for explaining two-dimensional map information.
3A and 3B are diagrams illustrating 3D map information.
4 is a diagram illustrating a method of operating an electronic device according to various embodiments of the present disclosure;
FIG. 5 is a diagram illustrating an operation of generating 3D map information of FIG. 4 .
6 and 7 are diagrams for explaining an operation of generating 3D map information of FIG. 4 .
FIG. 8 is a diagram illustrating a pose optimization operation of each street view image of FIG. 5 .
FIG. 9 is a diagram for explaining a pose optimization operation of each street view image of FIG. 5 .

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings.

Various embodiments provide an electronic device for providing visual localization (VL) based on outdoor 3D map information and an operating method thereof. According to various embodiments, the electronic device may provide visual localization in the outdoor environment by generating 3D map information on the outdoor environment. That is, the electronic device may perform positioning in the outdoor environment based on 3D map information on the outdoor environment. In this case, the electronic device may match the pose of the street view image required to generate the 3D map information based on the 2D map information. Through this, it is possible to minimize an error that occurs when the electronic device provides visual localization. For example, when cursoring is performed on 3D map information using 2D map information, distortion of a shape or size of an item related to the cursoring may be minimized. In addition, in providing augmented reality navigation (AR navigation) based on visual localization, point of interest (POI) (eg, shops, crosswalks, buildings, etc.) data of 2D map information may be utilized.

1 is a diagram illustrating an electronic device 100 according to various embodiments. 2 is a diagram for explaining two-dimensional map information. 3A and 3B are diagrams illustrating 3D map information.

Referring to FIG. 1 , an electronic device 100 according to various embodiments may use at least one of a communication module 110 , an input module 120 , an output module 130 , a memory 140 , and a processor 150 . may include In some embodiments, at least one of the components of the electronic device 100 may be omitted, and at least one other component may be added. In some embodiments, at least any two of the components of the electronic device 100 may be implemented as one integrated circuit. Here, the electronic device 100 may include at least one of a server and an electronic device. For example, the electronic device includes a smart phone, a mobile phone, a navigation system, a computer, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, a game console, It may include at least one of a wearable device, an Internet of things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, and a robot.

The communication module 110 may communicate with the external devices 181 and 183 in the electronic device 100 . The communication module 110 may establish a communication channel between the electronic device 100 and the external devices 181 and 183 and communicate with the external devices 181 and 183 through the communication channel. Here, the external devices 181 and 183 may include at least one of a satellite, a server, and an electronic device. For example, the electronic device includes a smartphone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcasting terminal, a PDA, a portable multimedia player (PMP), a tablet PC, a game console, a wearable device, an IoT device, a virtual reality (VR) device, It may include at least one of an AR device and a robot. The communication module 110 may include at least one of a wired communication module and a wireless communication module. The wired communication module may be connected to the external device 181 by wire to communicate via wire. The wireless communication module may include at least one of a short-range communication module and a long-distance communication module. The short-range communication module may communicate with the external device 181 in a short-distance communication method. For example, the short-range communication method may include Bluetooth, WiFi direct, or infrared data association (IrDA). The remote communication module may communicate with the external device 183 in a long-distance communication method. Here, the remote communication module may communicate with the external device 183 through the network 190 . For example, network 190 may include a cellular network, the Internet, or a computer network such as a local area network (LAN) or a wide area network (WAN), or the like.

The input module 120 may input a signal to be used in at least one component of the electronic device 100 . The input module 120 may include at least one of an input device configured to allow a user to directly input a signal to the electronic device 100 or a sensor device configured to generate a signal by sensing a surrounding environment. For example, the input device may include at least one of a microphone, a mouse, and a keyboard. In some embodiments, the sensor device may include at least one of a touch circuitry configured to sense a touch or a sensor circuit configured to measure the intensity of a force generated by the touch.

The output module 130 may output information to the outside of the electronic device 100 . The output module 130 may include at least one of a display device configured to visually output information and an audio output device capable of outputting information as an audio signal. For example, the display device may include at least one of a display, a hologram device, and a projector. For example, the display device may be implemented as a touch screen by being assembled with at least one of a touch circuit and a sensor circuit of the input module 120 . For example, the audio output device may include at least one of a speaker and a receiver.

The memory 140 may store various data used by at least one component of the electronic device 100 . For example, the memory 140 may include at least one of a volatile memory and a non-volatile memory. The data may include at least one program and input data or output data related thereto. The program may be stored in the memory 140 as software including at least one instruction, and may include at least one of an operating system, middleware, and an application.

The processor 150 may execute a program in the memory 140 to control at least one component of the electronic device 100 . Through this, the processor 150 may process data or perform an operation. In this case, the processor 150 may execute a command stored in the memory 140 .

According to various embodiments, the processor 150 may provide visual localization based on outdoor 3D map information. That is, the processor 150 may perform localization in the outdoor environment based on the outdoor 3D map information. To this end, 3D map information for the outdoor environment may be required. The processor 150 may generate 3D map information based on at least one of 2D map information and 2D image information for the outdoor environment. For example, the outdoor environment may be defined as a specific area. The processor 150 may store 3D map information in the memory 140 . In addition, the processor 150 may perform positioning based on the 3D map information, if necessary.

The processor 150 may obtain 2D map information for the outdoor environment from the external devices 181 and 183, for example, a server. In this case, the 2D map information may include a location of at least one object, for example, a global positioning system (GPS) location. For example, two-dimensional map information may be obtained as shown in FIG. 2 .

The processor 150 may acquire 2D image information through scanning of the outdoor environment. In this case, the 2D image information may include a plurality of street view images and pose information for each of the street view images. Here, the pose information may include a position, for example, a GPS position, and a pose, for example, 3-axis position values and 3-axis direction values. For example, the processor 150 may store the 2D image information in the memory 140 in advance and, if necessary, obtain the 2D image information from the memory 140 .

The processor 150 may generate 3D map information from the street view images. In this case, the processor 150 may generate 3D map information using the street view images using a structure from motion (SfM) algorithm. For example, 3D map information may be generated as shown in FIG. 3A or 3B .

The processor 150 may update pose information of the 3D map information based on the 2D map information. In this case, the processor 150 may detect at least one object from the 2D map information. For example, the processor 150 may detect a line outside the object from the 2D map information. In addition, the processor 150 may detect a region corresponding to the object from the 3D map information. Here, the processor 150 may detect a region in the 3D map information based on the location of the object in the 2D map information. For example, the processor 150 may detect points outside the area based on a point cloud of 3D map information. Through this, the processor 150 may update the pose of the 3D map information so that the area is matched with the object. In other words, the processor 150 may match the points outside the area in the 3D map information to the lines outside the object in the 2D map information. For example, the processor 150 may match an area of 3D map information to an object of 2D map information by using an iterative closet point (ICP) algorithm.

4 is a diagram illustrating a method of operating the electronic device 100 according to various embodiments of the present disclosure.

Referring to FIG. 4 , the electronic device 100 may acquire 2D map information and 2D image information for an outdoor environment in operation 410 . The processor 150 may obtain 2D map information for the outdoor environment from the external devices 181 and 183, for example, a server. In this case, the 2D map information may include a location of at least one object, for example, a GPS location. Meanwhile, the processor 150 may acquire 2D image information through scanning of the outdoor environment. In this case, the 2D image information may include a plurality of street view images and pose information for each of the street view images. Here, the pose information may include a position, for example, a GPS position, and a pose, for example, 3-axis position values and 3-axis direction values. For example, the processor 150 may store the 2D image information in the memory 140 in advance. In addition, the processor 150 may obtain 2D image information from the memory 140 .

In operation 420 , the electronic device 100 may generate 3D map information for an outdoor environment based on at least one of 2D map information and 2D image information. The processor 150 may generate 3D map information based on 2D image information. In this case, the processor 150 may generate 3D map information using the street view images using the SfM algorithm. This will be described later in more detail with reference to FIGS. 5, 6 and 7 .

FIG. 5 is a diagram illustrating an operation of generating 3D map information of FIG. 4 . 6 and 7 are diagrams for explaining an operation of generating 3D map information of FIG. 4 .

Referring to FIG. 5 , the electronic device 100 detects at least one object 610 from 2D map information in operation 510 . The processor 150 may detect the object 610 from the 2D map information. For example, the processor 150 may detect a line 611 outside the object 610 from the 2D map information as shown in FIG. 6 . Here, the processor 150 may obtain the location of the object 610 from the 2D map information.

The electronic device 100 may generate 3D map information based on 2D image information in operation 520 . The processor 150 may generate 3D map information by using the street view images of the 2D image information. In operation 530 , the electronic device 100 may detect the region 620 corresponding to the object 610 from the 3D map information. Here, the processor 150 may detect the region 620 corresponding to the object 610 in the 3D map information based on the location of the object 610 in the 2D map information. For example, the processor 150 may generate a point cloud 621 as 3D map information as shown in FIG. 6 , and detect points 623 outside the area based on the point cloud 621 . there is.

The electronic device 100 may optimize the pose of the region 620 so that the region 620 is matched to the object 610 in operation 540 . As shown in FIG. 7 , the processor 150 processes the point cloud 621 of the 3D map information to match the points 623 outside the area 620 to the line 611 outside the object 610 . can do it For example, the processor 150 may register the region 620 with the object 610 using the ICP algorithm. Here, the points 623 outside the region 620 may perform at least one of rotation or translation. This will be described later in more detail with reference to FIGS. 8 and 9 .

FIG. 8 is a diagram illustrating a pose optimization operation of each street view image of FIG. 5 . FIG. 9 is a diagram for explaining a pose optimization operation of each street view image of FIG. 5 .

Referring to FIG. 8 , the electronic device 100 may calculate an error between the object 610 and the region 620 in operation 810 . The processor 150 may calculate an error based on the distance between the object 610 and the region 620 . In this case, the processor 150 may calculate errors for each of the points 623 outside the region 620 and sum the errors.

For example, a line equation for points on the line 611 outside the object 610 may be defined as in [Equation 1] below. And, the distance from the points 623 outside the area 620 to the points on the line 611 outside the object 610 is in a distance function as shown in Equation 2 below. can be determined accordingly. Through this, the processor 150 generates an error from the points 623 outside the area 620 to the points on the line 611 outside the object 610 based on the error function shown in Equation 3 below. can be summed up. Here, the error function may be based on a least square method for the distance function. Here, the processor 150 may assign at least one weight to each of the points 623 outside the area 620 based on a predetermined criterion.

는 직선 방정식을 나타내고,

는 객체(610) 외곽의 선(611) 상의 점들에 대한 벡터를 나타내고,

는 변수 값을 나타내고,

는 법선 벡터(normal vector)를 나타낼 수 있다. here,

represents the linear equation,

represents a vector for points on the line 611 outside the object 610,

represents the variable value,

may represent a normal vector.

는 거리 함수를 나타내고,

은 상기 [수학식 1]을 기반으로 결정되는 선 벡터(line vector)를 나타내고,

는 영역(620) 외곽의 점(623)들에 대한 벡터를 나타내고,

및

와 도 9의 (a)에 도시된 바와 같은 관계에 있을 수 있으며, 이를 통해 상기 [수학식 2]와 같은 거리 함수가 도 9의 (b)를 기반으로 정의될 수 있다. here,

represents the distance function,

represents a line vector determined based on [Equation 1],

denotes a vector for points 623 outside the region 620,

and

and may have a relationship as shown in FIG.

는 오차 함수를 나타내고,

는 회전 행렬(rotation matrix)을 나타내고,

은 영역(620) 외곽의 점(623)들의 총 개수를 나타내고,

은 각 점(623)의 식별자를 나타내고,

는 가중치를 나타낼 수 있다. 예를 들면, 가중치는 영역(620) 외곽의 점(623)들 생성 시 시맨틱 세그멘테이션(semantic segmentation) 기법을 통해 얻을 수 있는 의미 정보(건물, 가로수, 신호등, 도로, 등)에 따라 그 중요도를 부과하거나, 영역(620) 외곽의 점(623)들을 생성하기 위해 사용된 특징점 개수, SfM 추정 오차에 따라 다르게 부과될 수 있다.here,

represents the error function,

represents a rotation matrix,

represents the total number of points 623 outside the area 620,

represents the identifier of each point 623,

may represent a weight. For example, the weight is assigned its importance according to semantic information (buildings, street trees, traffic lights, roads, etc.) that can be obtained through a semantic segmentation technique when the points 623 outside the area 620 are generated. Alternatively, different charges may be applied according to the number of feature points used to generate the points 623 outside the region 620 and the SfM estimation error.

The electronic device 100 may compare the error between the object 610 and the region 620 with a predetermined threshold in operation 820 . The processor 150 may determine whether the error is equal to or less than a threshold. For example, the threshold may be greater than or equal to zero. If the error is equal to or less than the threshold in operation 820 , the electronic device 100 returns to FIG. 4 and proceeds to operation 430 .

Meanwhile, if the error exceeds the threshold in operation 820 , the electronic device 100 may estimate the pose of the region 620 in the 3D map information in operation 830 . The processor 150 may estimate the pose of the region 620 in the 3D map information in a direction in which an error is reduced. That is, the processor 150 sets the point 623 outside the area 620 so that the distance from the points 623 outside the area 620 to the target points on the line 611 outside the object 610 is shortened. their location can be estimated. Thereafter, the electronic device 100 may return to FIG. 5 and proceed to operation 550 . Referring back to FIG. 5 , the electronic device 100 may update 3D map information and street view images based on the optimized pose in operation 550 . The processor 150 may update the 3D map information using the optimized pose. Then, the processor 150 may update at least one pose among the street view images by using the optimized pose. Thereafter, the electronic device 100 may return to FIG. 4 and proceed to operation 430 .

According to an embodiment, although not shown, after performing operation 550, the electronic device 100 may re-perform operations 530, 540, and 550. Through this, the electronic device 100 may re-detect the region 620 corresponding to the object 610 from the 3D map information in operation 530 . In operation 540 , the electronic device 100 may optimize the pose of the region 620 so that the region 620 is matched to the object 610 . Through this, the electronic device 110 may update the 3D map information and the street view images based on the optimized pose in operation 550 . In this case, the electronic device 100 may repeatedly perform operations 530 , 540 , and 550 until the error becomes less than or equal to the threshold in operation 820 of FIG. 8 .

Referring back to FIG. 4 , in operation 430 , the electronic device 100 may provide visual localization based on 3D map information.

According to an embodiment, the processor 150 may perform positioning in an outdoor environment based on 3D map information. For example, the processor 150 may perform positioning in response to a query image received from the external devices 181 and 183 . The processor 150 may receive a query image from the external devices 181 and 183 through the communication module 110 . The processor 150 may detect 3D map information for any one of the plurality of regions 510 based on the query image. In this case, the processor 150 may extract feature information from the query image through a deep learning model, and detect 3D map information using the feature information. In addition, the processor 150 may estimate the location of a point corresponding to the query image in the 3D map information.

According to another embodiment, the processor 150 may perform cursoring based on 3D map information. For example, the processor 150 may perform cursoring in response to a user's request. The processor 150 may receive a user's request for a specific area 620 in the 3D map information through the input module 120 . The processor 150 may determine the object 610 corresponding to the specific region 620 from the 2D map information, and detect depth information of the object. In addition, the processor 150 may estimate depth information of the specific region 620 from the 3D map information by using the depth information of the object 610 . In this case, since the specific region 620 is matched to the object 610 , the processor 150 may estimate depth information of the object 610 as depth information of the specific region 620 . Through this, the processor 150 may perform cursoring on the specific region 620 on the 3D map information by using the depth information of the specific region 620 .

According to another embodiment, in providing augmented reality navigation (AR navigation) based on visual localization, the processor 150 is a point of interest (POI) of two-dimensional map information (eg, a store, a crosswalk, buildings, etc.) can be used.

A method of operating the electronic device 100 according to various embodiments includes an operation of acquiring 2D map information for an outdoor environment and a plurality of street view images, and a 3D map using the 2D map information and the street view images. It may include an operation of generating information, and an operation of providing visual localization for the outdoor environment based on the 3D map information.

According to various embodiments, the generating of the 3D map information includes generating 3D map information from street view images, and updating a pose of the 3D map information based on the 2D map information. may include.

According to various embodiments, the operation of generating the 3D map information includes the operation of detecting the object 610 from the 2D map information, the operation of detecting an area corresponding to the object from the 3D map information, and the object 610 . The method further includes an operation of updating the pose of the 3D map information so that the region 620 is matched to the can do.

According to various embodiments, the operation of generating 3D map information may generate 3D map information from street view images using an SfM algorithm.

According to various embodiments, updating the pose of the 3D map information includes calculating an error between the object 610 and the region 620 and estimating a pose of the 3D map information so that the error is reduced. and updating the 3D map information with the estimated pose.

According to various embodiments, the operation of estimating the pose of the 3D map information is performed when an error exceeds a predetermined threshold.

According to various embodiments, the operation of detecting the area 620 corresponding to the object 610 may be repeated after the operation of updating to the estimated pose.

According to various embodiments, the updating of the pose of the 3D map information may update the pose of the 3D map information so that the region 620 is matched to the object 610 using an ICP algorithm.

According to various embodiments, the operation of providing the visual localization includes the operation of detecting depth information of the object 610 from the 2D map information, and the operation of detecting the depth information of the object 610 from the 3D map information using the detected depth information. ) may include an operation of estimating depth information of the region 620 corresponding to ), and an operation of providing visual localization by using the estimated depth information.

The electronic device 100 according to various embodiments may include a memory 140 and a processor 150 connected to the memory 140 and configured to execute at least one command stored in the memory.

According to various embodiments, the processor 150 obtains two-dimensional map information and a plurality of street view images for an outdoor environment, and generates three-dimensional map information by using the two-dimensional map information and the street view images. , based on the three-dimensional map information, may be configured to provide visual localization for the outdoor environment.

According to various embodiments, the processor 150 may be configured to generate 3D map information from street view images and update a pose of the 3D map information based on the 2D map information.

According to various embodiments, the processor 150 detects the object 610 from the 2D map information, detects the area 620 corresponding to the object 610 from the 3D map information, and the object 610 It may be configured to update the pose of the 3D map information so that the area 620 is matched to the , and update the pose of at least one of the street view images so that the area 620 is matched to the object 610 .

According to various embodiments, the processor 150 may be configured to generate 3D map information from street view images by using the SfM algorithm.

According to various embodiments, the processor 150 calculates an error between the object 610 and the region 620 , estimates a pose of the 3D map information so that the error is reduced, and applies the 3D map information to the 3D map information. , can be configured to update with the estimated pose.

According to various embodiments, the processor 150 may be configured to estimate a pose of the 3D map information when the error exceeds a predetermined threshold.

According to various embodiments, the processor 150 may be configured to re-detect the region 620 corresponding to the object 610 from the 3D map information after updating the 3D map information to the estimated pose. can

According to various embodiments, the processor 150 may be configured to update the pose of the 3D map information so that the region 620 is matched to the object 610 using an ICP algorithm.

According to various embodiments, the processor 150 detects depth information of the object 610 from the 2D map information, and uses the detected depth information to provide an area corresponding to the object 610 in the 3D map information. Estimate the depth information of 620 , and use the estimated depth information to provide visual localization.

Various embodiments of the present document may be implemented as a computer program including one or more instructions stored in a storage medium (eg, memory 140) readable by a computer device (eg, electronic device 100). can For example, the processor (eg, the processor 150 ) of the computer device may call at least one of one or more instructions stored from a recording medium and execute it. This enables the computer device to be operated to perform at least one function according to at least one command called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The computer-readable recording medium may be provided in the form of a non-transitory recording medium. Here, 'non-transitory' only means that the recording medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the recording medium and It does not distinguish between temporary storage cases.

A computer program according to various embodiments may be stored in a computer-readable recording medium in order to be combined with a computer device and execute a method for providing a visual localization in the computer device.

According to various embodiments, a method of providing visual localization includes an operation of acquiring two-dimensional map information for an outdoor environment and a plurality of street view images, and three-dimensional map information using the two-dimensional map information and the street view images. It may include an operation of generating , and an operation of providing visual localization for an outdoor environment based on 3D map information.

According to various embodiments, the electronic device 100 may provide visual localization in the outdoor environment by generating 3D map information for the outdoor environment. That is, the electronic device 100 may perform positioning in the outdoor environment based on 3D map information on the outdoor environment. At this time, by matching the poses of the street view image required for the electronic device 100 to generate the 3D map information based on the 2D map information, an error occurring when providing the visual localization may be minimized. For example, when cursoring is performed on 3D map information using 2D map information, distortion of a shape or size of an item related to the cursoring may be minimized.

It should be understood that the various embodiments of this document and the terms used therein are not intended to limit the technology described in this document to a specific embodiment, and include various modifications, equivalents, and/or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for like components. The singular expression may include the plural expression unless the context clearly dictates otherwise. In this document, expressions such as “A or B”, “at least one of A and/or B”, “A, B or C” or “at least one of A, B and/or C” refer to all of the items listed together. Possible combinations may be included. Expressions such as “first”, “second”, “first” or “second” can modify the corresponding components regardless of order or importance, and are only used to distinguish one component from another. It does not limit the corresponding components. When an (eg, first) component is referred to as being “(functionally or communicatively) connected” or “connected” to another (eg, second) component, that component is It may be directly connected to the component, or may be connected through another component (eg, a third component).

As used herein, the term “module” includes a unit composed of hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic block, component, or circuit. A module may be an integrally formed part or a minimum unit or a part of one or more functions. For example, the module may be configured as an application-specific integrated circuit (ASIC).

According to various embodiments, each component (eg, a module or a program) of the described components may include a singular or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to integration. According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

Claims (14)

A method of operating an electronic device, comprising:
acquiring two-dimensional map information for an outdoor environment and a plurality of street view images;
generating 3D map information by using the 2D map information and the street view images; and
and providing visual localization for the outdoor environment based on the 3D map information.
The method of claim 1, wherein the generating of the 3D map information comprises:
generating the 3D map information from the street view images; and
and updating the pose of the 3D map information based on the 2D map information.
The method of claim 2, wherein the generating of the 3D map information comprises:
detecting an object from the two-dimensional map information;
detecting an area corresponding to the object from the 3D map information;
updating the pose of the 3D map information so that the region matches the object; and
The method further comprising updating the pose of at least one of the street view images so that the region is registered with the object.
The method of claim 2, wherein the generating of the 3D map information comprises:
A method of generating the 3D map information from the street view images by using a structure from motion (SfM) algorithm.
The method of claim 3, wherein updating the pose of the 3D map information comprises:
calculating an error between the object and the region;
estimating the pose of the 3D map information so that the error is reduced; and
and updating the 3D map information with the estimated pose.
The method of claim 5, wherein estimating the pose of the 3D map information comprises:
if the error exceeds a predetermined threshold.
The method of claim 6, wherein the detecting of the region corresponding to the object comprises:
After updating with the estimated pose, the method is repeated.
The method of claim 3, wherein updating the pose of the 3D map information comprises:
A method of updating the pose of the 3D map information so that the region is matched to the object by using an iterative closet point (ICP) algorithm.
The method of claim 1 , wherein providing the visual localization comprises:
detecting depth information of an object from the two-dimensional map information;
estimating depth information of a region corresponding to the object from the 3D map information by using the detected depth information; and
and providing the visual localization by using the estimated depth information.
In an electronic device,
Memory; and
a processor coupled to the memory and configured to execute at least one instruction stored in the memory;
The processor is
Acquire two-dimensional map information and a plurality of street view images for the outdoor environment,
using the 2D map information and the street view images to generate 3D map information,
an apparatus configured to provide visual localization for the outdoor environment based on the three-dimensional map information.
The method of claim 10, wherein the processor comprises:
generate the 3D map information from the street view images,
an apparatus configured to update the pose of the 3D map information based on the 2D map information.
The method of claim 11 , wherein the processor comprises:
detecting an object from the two-dimensional map information,
detecting an area corresponding to the object from the 3D map information,
updating the pose of the 3D map information so that the region matches the object;
and update the pose of at least one of the street view images to match the region to the object.
The method of claim 10, wherein the processor comprises:
detecting depth information of an object from the two-dimensional map information,
estimating depth information of a region corresponding to the object from the 3D map information using the detected depth information,
and provide the visual localization using the estimated depth information.
A computer program stored in a computer-readable recording medium in combination with a computer device to execute a method for providing visual localization on the computer device,
The method of providing the visual localization,
acquiring two-dimensional map information for an outdoor environment and a plurality of street view images;
generating 3D map information by using the 2D map information and the street view images; and
and providing visual localization for the outdoor environment based on the three-dimensional map information.

Images