KR102675282B1 - Method and system for estimating absolute pose of road view images using automatically generated control features - Google Patents

Abstract

The present disclosure provides a method for estimating the absolute pose of a street view image, which is performed by at least one processor, comprising: receiving a plurality of control features associated with a specific area; Receiving a plurality of matching points obtained through feature matching between images and 3D absolute coordinate position information of at least one of the plurality of street view images based on the plurality of control features and the plurality of matching points; It includes estimating direction information, wherein each of the plurality of control features includes 3D absolute coordinate position information, and the plurality of control features includes at least one of a ground control point, a building control point, or a ground control line.

Description

Absolute pose estimation method and system for street view images using automatically generated control features {METHOD AND SYSTEM FOR ESTIMATING ABSOLUTE POSE OF ROAD VIEW IMAGES USING AUTOMATICALLY GENERATED CONTROL FEATURES}

The present disclosure relates to a method and system for estimating the absolute pose of a street view image, and specifically, a method for estimating the absolute pose of a street view image using a plurality of automatically generated control features including 3D absolute coordinate position information, and It's about the system.

As information technology develops, map information services are commercialized, and street view images are provided as an area of map information services. For example, a map information service provider can obtain images of a real space using a moving object on the ground and then provide the images taken at a specific point on an electronic map as a street view image.

However, street view images do not include precise absolute coordinate location information and posture information. In order to acquire 3D absolute coordinate location information for street view images captured using moving objects on the ground, a large number of markers are installed and measured over a wide range to establish a control point/control line. There is a problem in that it requires a lot of cost and effort to collect. In addition, even if control points/control lines are automatically collected using image processing technology, many mismatches occur between the collected control points/control lines and feature points included in the street view image, so accurate absolute coordinate location information and posture are not required. There is difficulty in estimating information.

The present disclosure provides a method for solving the above problems, a computer program stored in a recording medium, and a device (system).

The present disclosure may be implemented in various ways, including as a method, device (system), or computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, a method for estimating the absolute pose of a street view image, performed by at least one processor, includes receiving a plurality of control features associated with a specific area, a plurality of street view images associated with a specific area, Receiving a plurality of matching points obtained through feature matching between and estimating 3D absolute coordinate position information and direction information of at least one of the plurality of street view images based on the plurality of control features and the plurality of matching points. It includes a step, each of the plurality of control features includes 3D absolute coordinate position information, and the plurality of control features includes at least one of a ground control point, a building control point, or a ground control line.

In order to execute the method according to an embodiment of the present disclosure on a computer, a computer program stored in a computer-readable recording medium is provided.

According to an embodiment of the present disclosure, an information processing system includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and at least one The program receives a plurality of control features associated with a specific area, receives a plurality of matching points obtained through feature matching between a plurality of street view images associated with a specific area, and matches the plurality of control features and the plurality of matching points. and instructions for estimating 3D absolute coordinate position information and direction information of at least one of the plurality of street view images based on the plurality of control features, each of the plurality of control features including 3D absolute coordinate position information, and the plurality of control features , including at least one of a ground control point, a building control point, or a ground control line.

According to an embodiment of the present disclosure, 3D absolute coordinate location information and direction of the street view image are obtained using automatically acquired control features through feature matching between an aerial shot image including absolute coordinate information and a street view image. Information can be obtained automatically, quickly and accurately.

According to an embodiment of the present disclosure, 3D absolute coordinate position information and direction information of a high-quality street view image can be acquired using weights differently assigned to each type of a plurality of control features.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as “a person skilled in the art”) in the technical field to which this disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
1 is a diagram illustrating an example of a method for matching a 3D model and street view data according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a configuration in which an information processing system according to an embodiment of the present disclosure is connected to enable communication with a plurality of user terminals.
Figure 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure.
FIG. 4 is a block diagram illustrating a method for estimating 3D absolute coordinate location information and direction information of a street view image associated with a specific area according to an embodiment of the present disclosure.
Figure 5 is a diagram for explaining a method of determining the weight of a control feature according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating a method of estimating 3D absolute coordinate position information and direction information of a specific street view image using a ground control point according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating a method of estimating 3D absolute coordinate position information and direction information of a street view image using a building control point according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating a method of estimating 3D absolute coordinate position information and direction information of a street view image using a ground control line according to an embodiment of the present disclosure.
FIG. 9 is a diagram illustrating a method of estimating camera parameters for a street view image according to an embodiment of the present disclosure.
FIG. 10 is a diagram illustrating a specific example of a method for estimating camera parameters for a street view image according to an embodiment of the present disclosure.
FIG. 11 is a diagram illustrating an example of a loss function used when performing filtering according to an embodiment of the present disclosure.
Figure 12 is a flowchart illustrating an example of an absolute pose estimation method of a street view image according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In this disclosure, 'display' may refer to any display device associated with a computing device, e.g., any display device capable of displaying any information/data controlled by or provided by the computing device. can refer to.

In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or may refer to each of some components included in a plurality of A. .

FIG. 1 is a diagram illustrating an example of a method of matching a 3D model 110 and street view data 120 according to an embodiment of the present disclosure. The information processing system may acquire/receive the 3D model 110 and street view data 120 for a specific area.

The 3D model 110 may include 3D geometric information expressed in absolute coordinate positions and texture information corresponding thereto. Here, the location information included in the 3D model 110 may be information of higher accuracy than the location information included in the street view data 120. Additionally, the texture information included in the 3D model 110 may be of lower quality (eg, lower resolution) than the texture information included in the street view data 120. According to one embodiment, 3D geometric information expressed as an absolute coordinate position may be generated based on an aerial photograph taken of a specific area from above the specific area.

The 3D model (110) for a specific area includes a 3D building model (112), a digital elevation model (DEM) (114), a true ortho image (116) for a specific area, and a digital surface. It may include a digital surface model (DSM), road layout, road DEM, etc. As a specific example, the 3D model 110 for a specific area includes a digital surface model (DSM) containing geometric information about the ground of the specific area and an orthoimage 116 for the specific area corresponding thereto. It may be a model created based on, but is not limited to, this. In one embodiment, a precise orthoimage 116 of a specific area may be generated based on a plurality of aerial photos and the absolute coordinate location information and direction information of each aerial photo.

The street view data 120 may include absolute coordinate location information and direction information (i.e., image shooting direction information) for each of a plurality of street view images and a plurality of street view images captured at a plurality of nodes within a specific area. . Here, the location information and direction information included in the street view data 120 may be information of lower accuracy than the location information and direction information included in the 3D model 110, and the texture information included in the street view image is 3 It may be information of higher quality (eg, higher resolution) than the texture information included in the dimensional model 110. For example, the location information and direction information included in the street view data 120 may be location information and direction information obtained using GPS equipment when shooting a street view image at a node. Location information obtained using a vehicle's GPS device may have an error of about 5 to 10 meters.

The information processing system may perform map matching 130 between the 3D model 110 and street view data 120. Specifically, the information processing system may perform feature matching between texture information included in the 3D model 110 and a plurality of street view images included in the street view data 120. To perform map matching 130, the information processing system may convert at least some of the plurality of street view images included in the street view data 120 into a top view image. As a result of map matching 130, a plurality of map matching points/map matching lines 132 can be automatically extracted without user input. The map matching point/map matching line 132 is also referred to as a control feature. Each control feature may include 3D absolute coordinate position information and a visual descriptor.

The map matching point may represent a corresponding pair of a point in the street view image and a point in the 3D model 110. The type of map matching point may vary depending on the type of 3D model 110 used for map matching 130, the location of the point, etc. For example, map matching points are Ground Control Points (GCP), which are point correspondence pairs on the ground within a specific area, and Building Control Points (BCP), which are point correspondence pairs on buildings within a specific area. Alternatively, it may include at least one of structure control points, which are point correspondence pairs in structures within a specific area. Map matching points can be extracted not only from the ground, buildings, and structures described above, but also from street view images and arbitrary areas of the 3D model 110.

The map matching line may represent a corresponding pair of one line of the street view image and one line of the 3D model 110. The type of map matching line may vary depending on the type of 3D model 110 used for map matching 130, the location of the line, etc. For example, map matching lines include Ground Control Line (GCL), which is a corresponding pair of lines on the ground within a specific area, and Building Control Line (BCL), which is a corresponding pair of lines on buildings within a specific area. , it may include at least one of a structure control line, which is a corresponding pair of lines in a structure within a specific area, or a lane control line, which is a corresponding pair of lines in a lane within a specific area. Map matching lines can be extracted from the ground, buildings, structures, and lanes described above, as well as street view images and arbitrary areas of the 3D model 110.

Additionally, the information processing system may perform feature matching 150 between a plurality of street view images to extract a plurality of feature point correspondence sets 152. According to one embodiment, for robust feature matching, feature matching 150 between a plurality of street view images may be performed using at least a portion of the 3D model 110. For example, feature matching 150 between street view images can be performed using the 3D building model 112 included in the 3D model 110.

Then, the information processing system provides absolute coordinate position information and Direction information can be estimated (160). For example, the processor may estimate absolute coordinate position information and direction information for a plurality of street view images using a bundle adjustment technique (160). According to one embodiment, the estimated absolute coordinate position information and direction information 162 is information in an absolute coordinate system representing the 3D model 110, and may be a parameter of 6 degrees of freedom (DoF). The absolute coordinate location information and direction information 162 estimated through this process may be data with higher precision than the absolute coordinate location information and direction information included in the street view data 120. A detailed description of the method for estimating 3D absolute coordinate position information and direction information for a plurality of street view images (160) will be described in detail later with reference to FIGS. 4 to 12.

According to one embodiment, the information processing system is based on a control feature associated with a specific area and a matching point obtained through feature matching between a plurality of street view images associated with a specific area, and a three-dimensional image of the street view image. Absolute coordinate position information and direction information (i.e., 6-DoF pose) can be estimated. Here, the control feature can be automatically obtained without user input by performing map matching 130 between the 3D model 110 and the street view data 120, and each control feature includes 3D absolute coordinate location information and visual descriptor. (visual descriptor) may be included. Through this configuration, the 3D absolute coordinate location information and direction information of the street view image can be estimated with high accuracy.

Figure 2 is a schematic diagram showing a configuration in which the information processing system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals 210_1, 210_2, and 210_3. As shown, a plurality of user terminals 210_1, 210_2, and 210_3 may be connected to an information processing system 230 capable of providing a map information service through a network 220. Here, the plurality of user terminals 210_1, 210_2, and 210_3 may include terminals of users receiving a map information service. Additionally, the plurality of user terminals 210_1, 210_2, and 210_3 may be cars that capture street view images from nodes. In one embodiment, the information processing system 230 includes one or more server devices and/or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to providing map information services, etc. Alternatively, it may include one or more distributed computing devices and/or distributed databases based on cloud computing services.

The map information service provided by the information processing system 230 may be provided to the user through an application or web browser installed on each of the plurality of user terminals 210_1, 210_2, and 210_3. For example, the information processing system 230 may provide information corresponding to a street view image request, an image-based location recognition request, etc. received from the user terminals 210_1, 210_2, and 210_3 through an application or perform corresponding processing. You can.

A plurality of user terminals 210_1, 210_2, and 210_3 may communicate with the information processing system 230 through the network 220. The network 220 may be configured to enable communication between a plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of wireless networks such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include communication methods utilizing communication networks that the network 220 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.) as well as user terminals (210_1, 210_2, 210_3). ) may also include short-range wireless communication between

In Figure 2, the mobile phone terminal (210_1), tablet terminal (210_2), and PC terminal (210_3) are shown as examples of user terminals, but they are not limited thereto, and the user terminals (210_1, 210_2, 210_3) use wired and/or wireless communication. This may be any computing device capable of installing and executing an application or a web browser. For example, user terminals include AI speakers, smartphones, mobile phones, navigation, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, It may include wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, set-top boxes, etc. In addition, in Figure 2, three user terminals (210_1, 210_2, 210_3) are shown as communicating with the information processing system 230 through the network 220, but this is not limited to this, and a different number of user terminals are connected to the network ( It may be configured to communicate with the information processing system 230 through 220).

According to one embodiment, the information processing system 230 receives a plurality of control features associated with a specific area from the user terminals 210_1, 210_2, and 210_3 and a plurality of feature matching between a plurality of street view images associated with the specific area. The matching points can be received. Then, the information processing system 230 may estimate 3D absolute coordinate position information and direction information of at least one of the plurality of street view images based on the plurality of control features and the plurality of matching points. Thereafter, the information processing system 230 may transmit the 3D absolute coordinate position information and direction information of the estimated street view image to the user terminals 210_1, 210_2, and 210_3. In addition, the information processing system 230 may transmit various service-related data based on data created using the 3D absolute coordinate location information and direction information of the estimated street view image to the user terminals 210_1, 210_2, and 210_3. .

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing an application or a web browser and capable of wired/wireless communication, for example, the mobile phone terminal 210_1, tablet terminal 210_2 of FIG. 2, It may include a PC terminal (210_3), etc. As shown, the user terminal 210 may include a memory 312, a processor 314, a communication module 316, and an input/output interface 318. Similarly, information processing system 230 may include memory 332, processor 334, communication module 336, and input/output interface 338. As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. Additionally, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or to output information and/or data generated from the user terminal 210.

Memories 312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 312 and 332 are non-permanent mass storage devices such as read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It can be included. As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the user terminal 210 or the information processing system 230 as a separate persistent storage device that is distinct from memory. Additionally, the memories 312 and 332 may store an operating system and at least one program code (eg, code for an application installed and running on the user terminal 210).

These software components may be loaded from a computer-readable recording medium separate from the memories 312 and 332. This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, disk, tape, DVD/CD- It may include computer-readable recording media such as ROM drives and memory cards. As another example, software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into memory 312, 332 based on a computer program installed by files provided over the network 220 by developers or a file distribution system that distributes installation files for applications. It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336. For example, processors 314 and 334 may be configured to execute received instructions according to program codes stored in recording devices such as memories 312 and 332.

The communication modules 316 and 336 may provide a configuration or function for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and may provide a configuration or function for the user terminal 210 and/or information processing. The system 230 may provide a configuration or function for communicating with other user terminals or other systems (for example, a separate cloud system, etc.). In one example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as memory 312 (e.g., a control feature associated with a specific region and a plurality of data associated with a request for a street view image, etc.) may be transmitted to the information processing system 230 through the network 220 under the control of the communication module 316. Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 is transmitted through the communication module 316 of the user terminal 210 through the communication module 336 and the network 220. It may be received by the user terminal 210. For example, the user terminal 210 may receive data related to a street view image including 3D absolute coordinate location information and direction information from the information processing system 230.

The input/output interface 318 may be a means for interfacing with the input/output device 320. As an example, input devices may include devices such as cameras, keyboards, microphones, mice, etc., including audio sensors and/or image sensors, and output devices may include devices such as displays, speakers, haptic feedback devices, etc. You can. As another example, the input/output interface 318 may be a means for interfacing with a device that has components or functions for performing input and output, such as a touch screen, integrated into one. For example, the processor 314 of the user terminal 210 uses information and/or data provided by the information processing system 230 or another user terminal when processing instructions of a computer program loaded in the memory 312. A service screen, etc. constructed by doing so may be displayed on the display through the input/output interface 318. In FIG. 3 , the input/output device 320 is shown not to be included in the user terminal 210, but the present invention is not limited to this and may be configured as a single device with the user terminal 210. Additionally, the input/output interface 338 of the information processing system 230 may be connected to the information processing system 230 or means for interfacing with a device (not shown) for input or output that the information processing system 230 may include. It can be. In FIG. 3, the input/output interfaces 318 and 338 are shown as elements configured separately from the processors 314 and 334, but the present invention is not limited thereto, and the input/output interfaces 318 and 338 may be configured to be included in the processors 314 and 334. there is.

The user terminal 210 and information processing system 230 may include more components than those in FIG. 3 . However, there is no need to clearly show most prior art components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the input/output devices 320 described above. Additionally, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, if the user terminal 210 is a smartphone, it may include components generally included in a smartphone, such as an acceleration sensor, a gyro sensor, an image sensor, a proximity sensor, a touch sensor, Various components such as an illuminance sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and a vibrator for vibration may be implemented to be further included in the user terminal 210. According to one embodiment, the processor 314 of the user terminal 210 may be configured to operate an application that provides a map information service. At this time, code associated with the corresponding application and/or program may be loaded into the memory 312 of the user terminal 210.

While a program for an application providing a map information service is running, the processor 314 uses input devices such as a touch screen, a keyboard, a camera including an audio sensor and/or an image sensor, and a microphone connected to the input/output interface 318. It is possible to receive text, images, videos, voices, and/or actions input or selected through, and store the received text, images, videos, voices, and/or actions in the memory 312 or use the communication module 316 and It can be provided to the information processing system 230 through the network 220. For example, the processor 314 receives a user's input requesting a plurality of matching points between a plurality of control features for a specific area and a plurality of street view images of a specific area, and the communication module 316 and the network 220 ) can be provided to the information processing system 230.

The processor 314 of the user terminal 210 manages, processes, and/or stores information and/or data received from the input/output device 320, other user terminals, the information processing system 230, and/or a plurality of external systems. It can be configured to do so. Information and/or data processed by processor 314 may be provided to information processing system 230 via communication module 316 and network 220. The processor 314 of the user terminal 210 may transmit information and/or data to the input/output device 320 through the input/output interface 318 and output the information. For example, the processor 314 may display the received information and/or data on the screen of the user terminal.

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals 210 and/or a plurality of external systems. Information and/or data processed by the processor 334 may be provided to the user terminal 210 through the communication module 336 and the network 220.

FIG. 4 is a block diagram illustrating a method for estimating 3D absolute coordinate location information and direction information 490 of a street view image associated with a specific area according to an embodiment of the present disclosure. According to one embodiment, at least one processor (e.g., at least one processor of an information processing system and/or a user terminal) provides 3D absolute coordinate location information and direction information 490 of a street view image associated with a specific area. In order to estimate , input data 410 associated with a specific region may be received. Here, the input data 410 may refer to a first set of matching points associated with a specific street view image and a first set of control features corresponding thereto. Control features may include three-dimensional absolute coordinate position information and visual descriptors. For example, the control feature may include at least one of a ground control point associated with a specific point on the ground, a building control point associated with a specific point on the building, and a ground control line associated with a specific line on the ground.

According to one embodiment, the processor may determine (420) the weight of the first set of control features to estimate absolute coordinate location information and direction information of a specific street view image. Here, the weight of the first set of control features may be determined differently depending on the ground control point, building control point, or ground control line. A specific method of determining the weight of the control feature will be described in detail later with reference to FIG. 5.

According to one embodiment, the processor may estimate (430) 3D absolute coordinate position information and direction information of the first accuracy of a specific street view image based on the input data (410) and the weight of the control feature. Here, the 3D absolute coordinate position information and direction information of the first accuracy are absolute coordinates of lower accuracy than the accuracy of the 3D absolute coordinate position information and direction information of the second accuracy and the final 3D absolute coordinate position information and direction information 490. It may mean coordinate location information and direction information. Specifically, the processor projects a first set of control features onto a specific street view image to generate a first set of projection data, and uses the processor to estimate first accurate three-dimensional absolute coordinate position information and direction information of the specific street view image. It can be used to do so. For example, the processor may determine the first set of matching points associated with a particular street view image and the first set of matching points associated with a particular street view image based on the weights of each of the first set of matching points, the first set of projection data, and the first set of control features. To minimize errors between projection data, 3D absolute coordinate position information and direction information of a specific street view image with first accuracy may be estimated (430). Here, the processor may not estimate camera parameters when estimating the initial location information and direction information of a specific street view image.

According to one embodiment, the processor may perform first filtering (440) based on 3D absolute coordinate position information and direction information of the estimated first accuracy. Specifically, the processor may project a first set of control features onto a specific street view image based on the 3D absolute coordinate position information and direction information of the estimated first accuracy to generate a second set of projection data. The processor may then determine a second set of control features with some of the first set of control features removed based on the error between the first set of matching points associated with the particular street view image and the second set of projection data. there is. Filtering criteria (loss function) may differ depending on the type of control feature (GCP, BCP, GCL). Filtering criteria are described in detail later in FIG. 11. In this case, the processor may determine the second set of matching points by removing the matching point corresponding to the removed control feature from the first set of matching points. That is, the processor extracts data exceeding a predetermined threshold range from the first set of matching points and the second set of projection data, and based on this, the outlier value ( By performing first filtering to remove outliers (440), a second set of matching points and a second set of control features associated with a specific street view image can be obtained.

According to one embodiment, the processor may determine (450) an individual weight of the input data 410 based on the 3D absolute coordinate position information and direction information of the estimated first accuracy. Here, the input data 410 to which individual weights are applied may represent a second set of matching points and a second set of control features from which outliers have been removed by first filtering. To this end, the processor may calculate the covariance of the 3D absolute coordinate position information and direction information of the estimated first accuracy. Here, the covariance represents the reliability compared to the initial value of the 3D absolute coordinate position information and direction information of the estimated first accuracy, and when the reliability is calculated inversely by the error propagation equation, the second set of matching points and the second set of control features You can know the reliability of each. The final calculated reliability of each of the second set of matching points and the second set of control features may be used to determine individual weights of each of the second set of matching points and the second set of control features. In other words, the processor may calculate individual reliability for each of the second set of matching points and the second set of control features based on the reliability of the estimated first accuracy of the three-dimensional absolute coordinate position information and direction information. . And, based on the calculated individual reliability, individual weights for each of the second set of matching points and the second set of control features may be determined.

According to one embodiment, the processor estimates a second accuracy of three-dimensional absolute coordinate position information and direction information of a specific street view image based on the second set of matching points, the second set of control features, and the calculated individual weights. (460)You can. To this end, the processor may estimate camera parameters of the particular street view image and use these together with the second set of matching points, the second set of control features, and the calculated individual weights to determine a second accuracy of the particular street view image. 3D absolute coordinate position information and direction information can be estimated (460). By using camera parameters estimated according to geometric information, the 3D absolute coordinate position information and direction information of the street view image can be estimated more accurately. A specific method of estimating camera parameters of a specific street view image will be described in detail later with reference to FIGS. 9 and 10.

According to one embodiment, the processor may perform second filtering (470) based on the 3D absolute coordinate position information and direction information of the estimated second accuracy. The second filtering is performed according to the same/similar procedure as the first filtering. Specifically, the processor may project the second set of control features onto the specific street view image based on the 3D absolute coordinate position information and direction information of the estimated second accuracy to generate a third set of projection data. The processor may then determine a third set of control features with some of the second set of control features removed based on the error between the third set of projection data and the second set of matching points associated with the particular street view image. there is. In this case, the matching point corresponding to the removed control feature can be removed from the second set of matching points to determine the third set of matching points.

According to one embodiment, the processor may determine (480) whether the position information and direction information of the estimated 3D absolute coordinates of a specific street view image are below a predetermined threshold. For example, if the error between the second set of matching points associated with a particular street view image and the third set of projection data is less than or equal to a predetermined threshold, the processor may provide three-dimensional absolute coordinate position information and orientation information of a second accuracy. It can be determined using the final 3D absolute coordinate position information and direction information 490. As a specific example, as a predetermined threshold, the angle value between the straight line connecting the projection data and the matching point with the shooting point location information as the origin may be determined to be 6 degrees, but it is not limited to this. In another example, if the error between the second set of matching points associated with a particular street view image and the third set of projection data exceeds a predetermined threshold, the processor performs 450 until the error is below the predetermined threshold. Steps through 480 can be repeated.

Figure 5 is a diagram for explaining a method of determining the weight of a control feature according to an embodiment of the present disclosure. According to one embodiment, the weight of the control feature may be determined differently depending on the distance between the reference point of the control feature and a specific street view image. For example, the weight may be determined to be a value between 0 and 1 depending on the distance between the reference point of the control feature and a specific street view image. Here, the control feature may include 3D absolute coordinate position information. Additionally, the control feature may include at least one of a ground control point, a building control point, or a ground control line.

In one embodiment, the ground control points 520 and 530 may be assigned a lower weight as the distance between the street view image 510 and the ground control points 520 and 530 increases. This is because in the case of an area corresponding to an object that is far away based on the shooting location in the street view image, the image quality of the area is low, so there is a high probability that the matching point (feature point) and the ground control point are incorrectly matched. For example, referring to the planar image 512 of the street view image shown in FIG. 5, the image quality of the first area 522 that is far away from the shooting position (origin) is lower than that of the second area corresponding to a relatively close object. It can be seen that the image quality is lower than that of area 532. The first area 522 may correspond to the first ground control point 520, and the second area 532 may correspond to the second ground control point 530. In this case, the first ground control point 520, which is farther away than the second ground control point 530 based on the shooting position (origin) of the street view image 510, has a lower weight than the second ground control point 530. You can have

In one embodiment, the relative position between the shooting position (origin) and the ground control points 520 and 530 may be determined by the y-axis value in the planar image 512 of the street view image. For example, referring to FIG. 5, the y-axis value may increase toward the bottom of the image, which may mean getting closer to the shooting position (origin). Therefore, the larger the y-axis value, the more weight can be assigned to the ground control point.

In one embodiment, the building control point may be given a higher weight as the distance between the street view image and the building control point increases. This means that if a building is located close to the shooting location in the street view image, there is a high probability that the shooting location is a narrow road such as an alley, and in the alley, the building control point itself is created incorrectly, so the matching point (feature point) and building control This is because there is a high probability that the points were matched incorrectly.

In one embodiment, the weight of the building control point may be calculated by dividing the distance value at which the target building is located based on the shooting location by the maximum configurable distance value based on the shooting location. Here, the maximum settable distance based on the shooting position may be calculated as 100m to 200m, but is not limited thereto. If the maximum configurable distance value is 100m and the distance between the shooting location and the target building is 50m, the weight of the building control point may be determined to be 0.5.

In one embodiment, no separate weight may be assigned to the ground control line. For example, all ground control lines could be given a weight of 1.

Through this configuration, mismatching that occurs between a plurality of automatically collected control features and matching points included in the street view image can be effectively removed, providing high-quality 3D absolute coordinate location information for a specific street view image. and direction information can be obtained quickly and accurately.

FIG. 6 is a diagram illustrating a method of estimating 3D absolute coordinate position information and direction information of a specific street view image 630 using a ground control point according to an embodiment of the present disclosure. According to one embodiment, the control feature may include a ground control point 622. The ground control point can be obtained based on feature matching between the street view image and the aerial image 620 including 3D absolute coordinate location information, and the ground control point is 3 points associated with a specific point on the ground of the street view image. Dimensions may include absolute coordinate location information and visual descriptors. Here, a specific point on the ground of the street view image may be referred to as a matching point (feature point) obtained through feature matching between a plurality of street view images.

According to one embodiment, the information processing system provides three-dimensional absolute coordinate location information and Direction information can be estimated. For example, the information processing system may generate projection data 624 by projecting the ground control point 622 onto a specific street view image 630. Additionally, the information processing system can estimate 3D absolute coordinate position information and direction information so that the error between the matching point 612 of a specific street view image and the projection data 624 is minimized. The information processing system can obtain the final 3D absolute coordinate position information and direction information of the specific street view image 630 by repeatedly estimating the 3D absolute coordinate position information and direction information based on the estimation result. The specific method for this was described above with reference to FIG. 4.

In FIG. 6, it is shown that the 3D absolute coordinate location information and direction information of a specific street view image 630 are estimated using a pair of a ground control point 622 and a matching point 612, but the method is not limited to this. The 3D absolute coordinate location information and direction information of a specific street view image 630 can be estimated using a plurality of pairs of control features and matching points. At this time, the weight described in FIG. 5 is applied to each control feature so that the 3D absolute coordinate position information and direction information of the specific street view image 630 can be estimated.

FIG. 7 is a diagram illustrating a method of estimating 3D absolute coordinate location information and direction information of a street view image 730 using a building control point according to an embodiment of the present disclosure. According to one embodiment, the control feature may include a building control point 722. Building control points can be obtained based on feature matching between street view images and aerial images including 3D absolute coordinate location information, and building control points are 3D absolute coordinates associated with specific points on the building in the street view image. May include location information and visual descriptors. Here, a specific point on a building in the street view image may be referred to as a matching point (feature point) obtained through feature matching between a plurality of street view images. For example, the building control point 722 may be obtained based on feature matching between the street view image 710 and the aerial image 720 including 3D absolute coordinate location information.

According to one embodiment, the information processing system provides three-dimensional absolute coordinate location information and Direction information can be estimated. For example, the information processing system may generate projection data 724 by projecting the building control point 722 onto a specific street view image 730. Additionally, the information processing system can estimate 3D absolute coordinate position information and direction information so that the error between the matching point 712 of a specific street view image and the projection data 724 is minimized. The information processing system can obtain the final 3D absolute coordinate position information and direction information of the specific street view image 630 by repeatedly estimating the 3D absolute coordinate position information and direction information based on the estimation result. The specific method for this was described above with reference to FIG. 4.

In Figure 7, it is shown that the 3D absolute coordinate location information and direction information of a specific street view image 730 are estimated using a pair of a single building control point 722 and a matching point 712, but the method is not limited to this. 3D absolute coordinate position information and direction information of a specific street view image 730 can be estimated using a plurality of pairs of control features and matching points. At this time, the weight described in FIG. 5 is applied to each control feature so that the 3D absolute coordinate position information and direction information of the specific street view image 730 can be estimated.

FIG. 8 is a diagram illustrating a method of estimating 3D absolute coordinate position information and direction information of a street view image 830 using a ground control line according to an embodiment of the present disclosure. According to one embodiment, the control feature may include a ground control line 822. The ground control line can be obtained based on feature matching between the street view image and the aerial image including 3D absolute coordinate location information, and the ground control line is the 3D absolute coordinate associated with a specific line on the ground in the street view image. May include location information and visual descriptors. Here, a specific line on the ground of the street view image may be referred to as a matching line (feature line) obtained through feature matching between a plurality of street view images. For example, a matching line can be expressed as two matching points. For example, the ground control line 822 may be obtained based on feature matching between the street view image 810 and the aerial image 820 including 3D absolute coordinate location information.

According to one embodiment, the information processing system is based on the ground control line 822 and the matching line 812 in the specific street view image 830, three-dimensional absolute coordinate location information and Direction information can be estimated. For example, the information processing system may generate projection data 824 by projecting the ground control line 822 onto a specific street view image 830 . Additionally, the information processing system can estimate 3D absolute coordinate position information and direction information so that the error between the matching line 812 of the specific street view image 830 and the projection data 824 is minimized. For example, the information processing system may use the first normal vector (n1) on the surface connecting the shooting position (origin) and the projection data 824 and the second normal vector (n1) on the surface connecting the shooting position (origin) and the matching line 812. 3D absolute coordinate position information and direction information can be estimated so that the error between the normal vector (n2) is minimized. The information processing system can obtain the final 3D absolute coordinate position information and direction information of the specific street view image 830 by repeatedly estimating the 3D absolute coordinate position information and direction information based on the estimation result. The specific method for this was described above with reference to FIG. 4.

In Figure 8, it is shown that the 3D absolute coordinate position information and direction information of a specific street view image 830 are estimated using a pair of a ground control line 822 and a matching line 812, but the method is not limited to this. 3D absolute coordinate location information and direction information of a specific street view image 830 can be estimated using a plurality of pairs of control features and matching points. At this time, the weight described in FIG. 5 is applied to each control feature so that the 3D absolute coordinate position information and direction information of the specific street view image 830 can be estimated.

FIG. 9 is a diagram illustrating a method of estimating camera parameters for a street view image 900 according to an embodiment of the present disclosure. According to one embodiment, the information processing system may estimate camera parameters of the specific street view image 900 based on the 3D absolute coordinate position information and direction information estimation results of the specific street view image 900. Here, the specific street view image 900 may be a 360-degree panoramic image generated using equirectangular projection.

In one embodiment, the information processing system may estimate camera area parameters that define a boundary line for dividing a specific street view image 900 into six images (cam 1 to cam 6). Boundaries can be expressed as lines drawn on a sphere. Since the specific street view image 900, which is a 360-degree panoramic image, may be arbitrarily processed to be provided to the user without any sense of difference from the real world, the specific street view image 900 may not match the geometric characteristics of the panoramic image. In order to match the specific street view image 900 to the geometric characteristics of the panoramic image and improve the estimation quality of the 3D absolute coordinate position information and direction information for the specific street view image 900, the information processing system estimates camera area parameters. You can.

Specifically, the specific street view image 900, which is a 360-degree panoramic image, is assumed to be a stitched image of images captured by six cameras, and the area corresponding to each camera (e.g., cam 1 to cam 6) is It can be estimated. For example, by estimating four nodes (912, 914, 916, 918) associated with the area corresponding to the first camera (cam 1) and determining the edges (lines) connecting each of these nodes, the first You can define the area corresponding to the camera (cam 1). Similarly, a boundary line can be defined to divide each of the areas corresponding to the remaining five cameras (e.g., cam 2 to cam 5). In this case, as shown in FIG. 9, the boundary line 920 within the panoramic image may be redefined according to the geometric characteristics of the panoramic image. For example, as the boundary line 920 is redefined according to the geometric characteristics of the panoramic image, the boundary line of the image corresponding to cam 1 may be inconsistent with the boundary line of the images corresponding to cam 2 and cam 3 adjacent thereto.

According to one embodiment, the information processing system may determine camera distortion parameters for converting six images (eg, cam 1 to cam 6) into undistorted flat images. The information processing system can use the camera distortion parameters for conversion to a non-distorted flat image along with the camera area parameters to estimate the 3D absolute coordinate position information and direction information of the street view image.

FIG. 10 is a diagram illustrating a specific example of a method for estimating camera parameters for a street view image according to an embodiment of the present disclosure. According to one embodiment, the information processing system can estimate camera area parameters of a street view image that is a 360-degree panoramic image. The first example 1010 shows an example in which, when dividing a street view image into 6 images, the FOV (Field of View) of each image is divided to 90 degrees. For example, the area corresponding to 45 degrees left, 45 degrees right, 45 degrees above, and 45 degrees below based on the center of the street view image, which is a 360-degree panoramic image, can be defined as the front image (Front). Also, the left image (Left) can be defined so that the FOV is 90 degrees based on a position rotated 90 degrees from the front to the left. Similarly, right image (Right), rear image (Back), upper image (Up), and lower image (Down) can be defined. At this time, each image may include four nodes corresponding to a rectangular grid. For example, the front image may include four nodes (1012, 1014, 1016, and 1018) corresponding to a rectangular grid.

The second example 1020 shows an example of the result of estimating camera area parameters to match the street view image to the geometric features of the panoramic image. For example, as described with reference to FIG. 9, camera area parameters that define a boundary line for dividing a street view image into six images can be estimated. For example, the area of the front image (Front) defined based on the estimated camera area parameter may include four nodes (1022, 1024, 1026, and 1028) corresponding to the distorted square.

Through this configuration, the street view image is matched to the geometric features of the panoramic image, and based on this, the 3D absolute coordinate location information and direction information for the street view image are estimated, providing high quality 3D absolute coordinate location information and direction. Information can be obtained.

FIG. 11 is a diagram illustrating an example of a loss function used when performing filtering according to an embodiment of the present disclosure. According to one embodiment, the loss function may be used as a standard for determining outliers based on the error between the matching point of the street view image and the projection data of the control feature. Here, the X-axis of the depicted graph is associated with the size of the error between the matching point of the street view image and the projection data of the control feature. Additionally, the Y axis of the graph is associated with the weight given to the corresponding control feature.

According to one embodiment, the loss function used to remove outliers included in the control feature may have a different scale depending on the type of control feature. For example, a loss function with a larger scale than the ground control point and building control point may be used for the ground control line.

The first example 1110 shows an example of a loss function with a small scale (e.g., scale: 1). For example, the first example 1110 may represent an example of a loss function for a ground control point and a building control point. In the case of the first example 1110, data whose error size is greater than or equal to 1 may be determined to be an outlier and removed.

The second example 1120 shows an example of a loss function with a large scale (eg, scale: 2). For example, the second example 1120 may represent an example of a loss function for a ground control line. In the case of the second example 1120, data with an error size of 2 or more may be determined to be an outlier and removed. Because the ground control line has a larger error than the ground control point and building control point due to the nature of the line, a loss function with a larger scale than the ground control point and building control point can be used for the ground control line.

Accordingly, in the case of a ground control line using a large-scale loss function, it is judged as normal data even if the error between the matching point of the street view image and the projection data of the control feature is relatively larger than the ground control point or building control point. It can be. On the other hand, in the case of ground control points and building control points that use a small-scale loss function, even if the error between the matching point of the street view image and the projection data of the control feature is relatively smaller than the ground control line, it can be judged as normal data. You can.

Figure 11 shows an example in which the scale value of the loss function is set to 1 or 2, but this is for convenience of explanation, and the scale value is not limited to this.

FIG. 12 is a flowchart illustrating an example of a method 1200 for estimating the absolute pose of a street view image according to an embodiment of the present disclosure. Method 1200 may be initiated by a processor (eg, at least one processor of an information processing system and/or a user terminal) receiving a plurality of control features associated with a specific region (S1210). Then, the processor may receive a plurality of matching points obtained through feature matching between a plurality of street view images related to a specific area (S1220). Here, each of the plurality of control features includes 3D absolute coordinate position information, and the plurality of control features may include at least one of a ground control point, a building control point, or a ground control line.

According to one embodiment, the plurality of control features may include 3D absolute coordinate location information and a visual descriptor associated with a specific point of the street view image. In one embodiment, the plurality of control features includes a plurality of ground control points, and each ground control point may include 3D absolute coordinate position information and a visual descriptor associated with a specific point on the ground. In another embodiment, the plurality of control features may include a plurality of building control points, and each building control point may include 3D absolute coordinate location information and a visual descriptor associated with a specific point in the building. In another embodiment, the plurality of control features may include a plurality of ground control lines, each ground control line including a visual descriptor and three-dimensional absolute coordinate position information associated with a particular line on the ground.

According to one embodiment, the processor may estimate 3D absolute coordinate position information and direction information of at least one of the plurality of street view images based on the plurality of control features and the plurality of matching points (S1230).

Specifically, 3D absolute coordinate position information and direction information with first accuracy can be estimated for a specific street view image included in a plurality of street view images. For example, the processor may identify a first set of matching points associated with a specific street view image among a plurality of matching points. Additionally, the processor may identify the first set of control features that correspond to the first set of matching points among the plurality of control features. Additionally, the processor may project a first set of control features onto a particular street view image to generate a first set of projection data. Then, the processor may estimate first accuracy three-dimensional absolute coordinate position information and direction information of the specific street view image, based on the first set of matching points and the first set of projection data. Here, the 3D absolute coordinate position information and direction information of the first accuracy may be estimated so that the error between the first set of matching points and the first set of projection data is minimized.

According to one embodiment, estimating 3D absolute coordinate position information and direction information of at least one of the plurality of street view images may further include determining a weight of each control feature of the first set. In one embodiment, when the control feature is a ground control point, the processor may assign a lower weight as the distance between a specific street view image and the ground control point increases. In another embodiment, when the control feature is a building control point, the processor may assign a higher weight as the distance between a specific street view image and the building control point increases.

Additionally, the processor may perform first filtering based on 3D absolute coordinate position information and direction information of the estimated first accuracy. Specifically, the processor may project a first set of control features onto a specific street view image based on the 3D absolute coordinate position information and direction information of the estimated first accuracy to generate a second set of projection data. Then, the processor may determine a second set of control features from which some of the first set of control features are removed based on an error between the first set of matching points and the second set of projection data. Then, the processor may determine a second set of matching points by removing matching points corresponding to the removed control features from the first set of matching points. In one embodiment, when removing some of the first set of control features, a loss function of a larger scale than the ground control points and building control points may be used for the ground control lines. Therefore, in the case of a ground control line using a large-scale loss function, even if the error between the matching point of the street view image and the projection data of the control feature is relatively larger than the ground control point or building control point, it can be judged as normal data. The probability may be high.

Additionally, the processor may estimate 3D absolute coordinate position information and direction information of a second accuracy based on the results of the estimated 3D absolute coordinate position information and direction information of the first accuracy. At this time, the second accuracy may be higher than the first accuracy. Specifically, the processor may determine individual weights for each of the second set of matching points and the second set of control features. For example, the processor may calculate individual reliability for each of the second set of matching points and the second set of control features based on the reliability of the first accuracy 3D absolute coordinate position information and direction information. Then, the processor may determine individual weights for each of the second set of matching points and the second set of control features based on the calculated individual reliability. The processor may then estimate camera parameters of the particular street view image based on the second set of matching points, the second set of control features, and the individual weights. For example, the processor may estimate camera region parameters that define boundaries for dividing a particular street view image into six images. Here, the specific street view image may be a 360-degree panoramic image created using equirectangular projection. Additionally, the processor can determine camera distortion parameters to convert the six images into an undistorted planar image. And, the processor may estimate a second accuracy of three-dimensional absolute coordinate position information and orientation information of the specific street view image based on the second set of matching points, the second set of control features, individual weights, and camera parameters. .

Additionally, the processor may perform second filtering based on the 3D absolute coordinate position information and direction information of the estimated second accuracy. Specifically, the processor may project the second set of control features onto the specific street view image based on the 3D absolute coordinate position information and direction information of the estimated second accuracy to generate a third set of projection data. And, the processor may determine a third set of control features from which some of the control features of the second set are removed based on an error between the second set of matching points and the third set of projection data. Then, the processor may determine a third set of matching points by removing matching points corresponding to the removed control features from the second set of matching points.

Additionally, the processor may project a third set of control features onto a particular street view image to generate a fourth set of projection data. And, it may be determined whether the errors between the third set of matching points and the fourth set of projection data are all less than or equal to a predetermined threshold. In one embodiment, when it is determined that the errors between the third set of matching points and the fourth set of projection data are all less than or equal to a predetermined threshold, the processor specifies three-dimensional absolute coordinate position information and orientation information of a second accuracy. It can be determined using the final 3D absolute coordinate location information and direction information of the street view image. On the other hand, when it is determined that the errors between the third set of matching points and the fourth set of projection data all exceed a predetermined threshold, the processor estimates the three-dimensional absolute coordinate position information and direction information of the specific street view image. The steps can be repeated.

The flowchart of FIG. 12 and the above description are only examples, and the scope of the present disclosure is not limited thereto. For example, at least one step may be added/changed/deleted, or the order of each step may be changed.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by a person skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: 3D model
112: 3D building model 114: Digital elevation model 116: Orthoimagery
120: Street view data
130: Map matching 132: Map matching point/line
150: Feature matching 152: Feature point correspondence set
160: Estimation of location information and direction information
162: Absolute coordinate position information and direction information

Claims (20)

In the absolute pose estimation method of a street view image, performed by at least one processor,
Receiving a plurality of control features associated with a specific region;
Receiving a plurality of matching points obtained through feature matching between a plurality of street view images associated with the specific area; and
Estimating 3D absolute coordinate position information and direction information of at least one of the plurality of street view images based on the plurality of control features and the plurality of matching points.
Including,
Each of the plurality of control features includes three-dimensional absolute coordinate position information,
The plurality of control features include at least one of a ground control point, a building control point, or a ground control line,
The plurality of street view images include a specific street view image,
the plurality of control features includes a first set of control features corresponding to the specific street view image,
The 3D absolute coordinate location information and direction information of the specific street view image are estimated based on the plurality of matching points, the first set of control features, and weights associated with each of the first set of control features,
A method for estimating an absolute pose of a street view image, wherein at least some of the weights associated with each of the control features of the first set are different from each other. According to paragraph 1,
The step of estimating 3D absolute coordinate location information and direction information of at least one of the plurality of street view images includes:
Identifying the first set of matching points associated with the specific street view image among a plurality of matching points at all times;
identifying the first set of control features among the plurality of control features that correspond to the first set of matching points;
projecting the first set of control features onto the specific street view image to generate a first set of projection data; and
Based on the first set of matching points and the first set of projection data, estimating first accuracy three-dimensional absolute coordinate position information and direction information of the specific street view image.
An absolute pose estimation method including. According to paragraph 2,
The 3D absolute coordinate position information and direction information of the first accuracy are estimated so that the error between the first set of matching points and the first set of projection data is minimized. According to paragraph 2,
The step of estimating 3D absolute coordinate location information and direction information of at least one of the plurality of street view images includes:
determining a weight for each of the first set of control features.
It further includes,
The three-dimensional absolute coordinate position information and direction information of the first accuracy of the specific street view image are based on the weights of each of the first set of matching points, the first set of projection data, and the first set of control features. Estimated absolute pose estimation method for street view images. According to paragraph 4,
Determining the weight of each control feature of the first set includes:
If the control feature is a ground control point, assigning a lower weight as the distance between the specific street view image and the ground control point increases.
Absolute pose estimation method for street view images, including. According to paragraph 4,
Determining the weight of each control feature of the first set includes:
If the control feature is a building control point, assigning a higher weight as the distance between the specific street view image and the building control point increases.
Absolute pose estimation method for street view images, including. According to paragraph 2,
Based on the estimated first accurate three-dimensional absolute coordinate position information and direction information, projecting the first set of control features onto a specific street view image to generate a second set of projection data;
based on an error between the first set of matching points and the second set of projection data, determining a second set of control features with some of the first set of control features removed; and
determining a second set of matching points by removing matching points corresponding to the removed control features from the first set of matching points.
An absolute pose estimation method for street view images, further comprising: In clause 7,
A method for absolute pose estimation of a street view image, wherein when removing some of the control features of the first set, a loss function with a larger scale than the ground control point and the building control point is used for the ground control line. In clause 7,
determining individual weights for each of the second set of matching points and the second set of control features;
estimating camera parameters of the specific street view image based on the second set of matching points, the second set of control features, and the individual weights; and
Estimating a second accuracy three-dimensional absolute coordinate position information and orientation information of the specific street view image based on the second set of matching points, the second set of control features, the individual weights, and the camera parameters.
It further includes,
The second accuracy is a higher accuracy than the first accuracy. According to clause 9,
The method for determining the individual weights is:
calculating individual reliability for each of the second set of matching points and the second set of control features, based on the reliability of the first accurate 3D absolute coordinate position information and direction information; and
determining individual weights for each of the second set of matching points and the second set of control features based on the calculated individual reliability.
Absolute pose estimation method for street view images, including. According to clause 9,
The step of estimating the camera parameters is,
Estimating camera area parameters defining a boundary line for dividing the specific street view image into six images.
Absolute pose estimation method for street view images, including. According to clause 11,
A method for estimating the absolute pose of a street view image, wherein the specific street view image is a 360-degree panoramic image generated by equirectangular projection. According to clause 11,
The step of estimating the camera parameters is,
Determining camera distortion parameters for converting the six images into undistorted flat images
An absolute pose estimation method for street view images, further comprising: According to clause 9,
Based on the estimated second accuracy of three-dimensional absolute coordinate position information and direction information, projecting the second set of control features onto a specific street view image to generate a third set of projection data;
based on an error between the second set of matching points and the third set of projection data, determining a third set of control features with some of the second set of control features removed; and
determining a third set of matching points by removing matching points corresponding to the removed control features from the second set of matching points.
An absolute pose estimation method for street view images, further comprising: According to clause 14,
generating a fourth set of projection data by projecting the third set of control features onto a specific street view image; and
determining whether errors between the third set of matching points and the fourth set of projection data are all less than or equal to a predetermined threshold.
An absolute pose estimation method for street view images, further comprising: According to paragraph 1,
The plurality of control features include a plurality of ground control points,
An absolute pose estimation method for street view images, wherein each ground control point includes 3D absolute coordinate location information and a visual descriptor associated with a specific point on the ground. According to paragraph 1,
The plurality of control features includes a plurality of building control points,
An absolute pose estimation method for street view images, where each building control point includes 3D absolute coordinate location information and a visual descriptor associated with a specific point in the building. According to paragraph 1,
The plurality of control features include a plurality of ground control lines,
A method for absolute pose estimation of street view images, wherein each ground control line includes three-dimensional absolute coordinate position information and a visual descriptor associated with a specific line on the ground. A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1 to 18 on a computer. As an information processing system,
communication module;
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory
Including,
The at least one program is,
Receive a plurality of control features associated with a specific region,
Receive a plurality of matching points obtained through feature matching between a plurality of street view images associated with the specific area,
Comprising instructions for estimating 3D absolute coordinate position information and direction information of at least one of the plurality of street view images based on the plurality of control features and the plurality of matching points,
Each of the plurality of control features includes three-dimensional absolute coordinate position information,
The plurality of control features include at least one of a ground control point, a building control point, or a ground control line,
The plurality of street view images include a specific street view image,
the plurality of control features includes a first set of control features corresponding to the specific street view image,
The 3D absolute coordinate location information and direction information of the specific street view image are estimated based on the plurality of matching points, the first set of control features, and weights associated with each of the first set of control features,
and wherein at least some of the weights associated with each of the first set of control features are different from each other.

Images