KR20240064332A - Method and system for automatically acquiring ground feature point matching points between road view images using 3d models - Google Patents

Abstract

The present disclosure relates to a method of acquiring matching pairs of feature points between street view images using a 3D model, which is performed by at least one processor. A method of acquiring feature point matching pairs between street view images using a 3D model includes the steps of receiving a 3D model for a specific area containing 3D geometric information expressed as an absolute position, and Receiving a first street view image - The first street view image includes a first building - Receiving a second street view image taken at a second node within a specific area - The second street view image includes the first building A building is included - and acquiring a plurality of feature point matching pairs based on the first street view image, the second street view image, and the 3D model.

Description

Method and system for automatically acquiring feature point matching pairs between street view images using a 3D model {METHOD AND SYSTEM FOR AUTOMATICALLY ACQUIRING GROUND FEATURE POINT MATCHING POINTS BETWEEN ROAD VIEW IMAGES USING 3D MODELS}

The present disclosure relates to a method and system for automatically acquiring feature point matching pairs between street view images using a 3D model. Specifically, information included in the 3D model when performing feature matching between street view images captured on the ground. It relates to a method and system for more accurately acquiring feature point matching pairs by using .

As information technology develops, content provision services become widespread, and content expression methods are also changing from two-dimensional content to three-dimensional content. 3D content can realistically express the object of expression, so it is widely used not only in entertainment fields such as movies and games, but also in map information services such as navigation and autonomous driving services. However, because it takes a lot of cost and effort to produce 3D content, research on technology for converting already existing 2D content into 3D content is being actively conducted.

Meanwhile, as an area of map information service, street view service is provided. The street view image in the street view service is an omnidirectional panoramic image captured at a specific location and can contain more information compared to a regular camera image with a narrow angle of view. Therefore, when producing 3D content based on street view images, it is more effective because it is possible to produce more 3D content with fewer images than using images captured using a general camera. However, in the case of panoramic street view images, which are generated using equirectangular projection, geometric distortion may occur depending on the location of the object area included in the street view image, making it difficult to produce 3D content based on this. .

The present disclosure provides a method for solving the above problems, a computer program stored in a recording medium, and a device (system).

The present disclosure may be implemented in various ways, including as a method, device (system), or computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, a method of acquiring feature point matching pairs between street view images using a 3D model, which is performed by at least one processor, is performed for a specific area including 3D geometric information expressed as an absolute position. Receiving a 3D model, receiving a first street view image captured at a first node within a specific area - the first street view image includes a first building -, receiving a first street view image captured at a second node within a specific area 2. Receiving a street view image - the second street view image includes the first building - and acquiring a plurality of feature point matching pairs based on the first street view image, the second street view image, and the 3D model. Includes.

In order to execute the method according to an embodiment of the present disclosure on a computer, a computer program stored in a computer-readable recording medium is provided.

According to an embodiment of the present disclosure, an information processing system includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory. At least one program receives a 3D model for a specific area including 3D geometric information expressed as an absolute position, receives a first street view image taken at a first node within the specific area, and - a first street view image. The review image includes a first building -, a second street view image taken from a second node within a specific area is received, - the second street view image includes a first building, -, the first street view image, the second street view image is received. Includes commands for acquiring a plurality of feature point matching pairs based on the review image and the 3D model.

According to an embodiment of the present disclosure, the quality of feature matching for a geometrically distorted image can be improved by automatically acquiring a feature point matching pair associated with a matching target building between street view data.

According to an embodiment of the present disclosure, by using a 3D model including 3D geometric information expressed in absolute position, the same matching target building is determined among numerous buildings with high appearance similarity, and feature point matching pairs for the building are automatically generated. This can reduce the cost and effort of acquiring matching pairs of feature points for each of multiple buildings included in the street view data.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as “a person skilled in the art”) in the technical field to which this disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.
1 is a diagram illustrating an example of a method for matching a 3D model and street view data according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram showing a configuration in which an information processing system according to an embodiment of the present disclosure is connected to enable communication with a plurality of user terminals.
Figure 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure.
FIG. 4 is a diagram illustrating an example of determining a matching candidate area among a plurality of buildings included in a street view image captured from the ground according to an embodiment of the present disclosure.
FIG. 5 is a diagram illustrating an example of a process of performing feature matching based on a first matching candidate area in a first street view image and a second matching candidate area in a second street view image according to an embodiment of the present disclosure.
FIG. 6 is a diagram illustrating an example of a plurality of feature points detected from a planar image of a matching target building according to an embodiment of the present disclosure.
FIG. 7 is a diagram illustrating an example of a matching result between a first planar image and a second planar image according to an embodiment of the present disclosure.
FIG. 8 is a diagram illustrating an example of a feature point matching result between street view images acquired based on matching results for each of a plurality of matching target buildings according to an embodiment of the present disclosure.
Figure 9 is a diagram illustrating an example of selecting a pair of street view images that commonly include the same building based on a street view image map.
FIG. 10 is a flowchart illustrating an example of a method for acquiring matching pairs of feature points between street view images using a 3D model according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, etc. In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In this disclosure, 'display' may refer to any display device associated with a computing device, e.g., any display device capable of displaying any information/data controlled by or provided by the computing device. can refer to.

In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or may refer to each of some components included in a plurality of A. .

FIG. 1 is a diagram illustrating an example of a method of matching a 3D model 110 and street view data 120 according to an embodiment of the present disclosure. The information processing system may acquire/receive the 3D model 110 and street view data 120 for a specific area.

The 3D model 110 may include 3D geometric information expressed in absolute coordinate positions and texture information corresponding thereto. Here, the location information included in the 3D model 110 may be information of higher accuracy than the location information included in the street view data 120. Additionally, the texture information included in the 3D model 110 may be of lower quality (eg, lower resolution) than the texture information included in the street view data 120. According to one embodiment, 3D geometric information expressed as an absolute coordinate position may be generated based on an aerial photograph taken of a specific area from above the specific area.

The 3D model (110) for a specific area includes a 3D building model (112), a digital elevation model (DEM) (114), a true ortho image (116) for a specific area, and a digital surface. It may include a digital surface model (DSM), road layout, road DEM, etc. As a specific example, the 3D model 110 for a specific area includes a digital surface model (DSM) containing geometric information about the ground of the specific area and an orthoimage 116 for the specific area corresponding thereto. It may be a model created based on, but is not limited to, this. In one embodiment, a precise orthoimage 116 of a specific area may be generated based on a plurality of aerial photos and the absolute coordinate location information and direction information of each aerial photo.

The street view data 120 may include a plurality of street view images captured at a plurality of nodes within a specific area and absolute coordinate location information for each of the plurality of street view images. Here, the location information included in the street view data 120 may be information of lower accuracy than the location information included in the 3D model 110, and the texture information included in the street view image is included in the 3D model 110. It may be information of higher quality (e.g., higher resolution) than the included texture information. For example, the location information included in the street view data 120 may be location information obtained using a GPS device when a node captures a street view image. Location information obtained using a vehicle's GPS equipment may have an error of about 5 to 10 meters. Additionally, street view data may include direction information (i.e., image shooting direction information) for each of a plurality of street view images.

The information processing system may perform map matching 130 between the 3D model 110 and street view data 120. Specifically, the information processing system may perform feature matching between texture information included in the 3D model 110 and a plurality of street view images included in the street view data 120. To perform map matching 130, the information processing system may convert at least some of the plurality of street view images included in the street view data 120 into a top view image. As a result of map matching 130, a plurality of map matching points/map matching lines 132 can be extracted.

The map matching point may represent a corresponding pair of a point in the street view image and a point in the 3D model 110. The type of map matching point may vary depending on the type of 3D model 110 used for map matching 130, the location of the point, etc. For example, map matching points are Ground Control Points (GCP), which are point correspondence pairs on the ground within a specific area, and Building Control Points (BCP), which are point correspondence pairs on buildings within a specific area. Alternatively, it may include at least one of structure control points, which are point correspondence pairs in structures within a specific area. Map matching points can be extracted not only from the ground, buildings, and structures described above, but also from street view images and arbitrary areas of the 3D model 110.

The map matching line may represent a corresponding pair of one line of the street view image and one line of the 3D model 110. The type of map matching line may vary depending on the type of 3D model 110 used for map matching 130, the location of the line, etc. For example, map matching lines include Ground Control Line (GCL), which is a corresponding pair of lines on the ground within a specific area, and Building Control Line (BCL), which is a corresponding pair of lines on buildings within a specific area. , it may include at least one of a structure control line, which is a corresponding pair of lines in a structure within a specific area, or a lane control line, which is a corresponding pair of lines in a lane within a specific area. Map matching lines can be extracted from the ground, buildings, structures, and lanes described above, as well as street view images and arbitrary areas of the 3D model 110.

Additionally, the information processing system may perform feature matching 150 between a plurality of street view images to extract a plurality of feature point correspondence sets 152. According to one embodiment, for robust feature matching, feature matching 150 between a plurality of street view images may be performed using at least a portion of the 3D model 110. For example, feature matching 150 between street view images can be performed using the 3D building model 112 included in the 3D model 110. In the present disclosure, a method of acquiring feature point matching pairs between street view images using a 3D model will be described in detail later with reference to FIGS. 4 to 10.

Then, the information processing system provides absolute coordinate position information and Direction information can be estimated (160). For example, the processor may estimate absolute coordinate position information and direction information for a plurality of street view images using a bundle adjustment technique (160). According to one embodiment, the estimated absolute coordinate position information and direction information 162 is information in an absolute coordinate system representing the 3D model 110, and may be a parameter of 6 degrees of freedom (DoF). The absolute coordinate location information and direction information 162 estimated through this process may be data with higher precision than the absolute coordinate location information and direction information included in the street view data 120.

According to an embodiment of the present disclosure, the quality of feature matching for a geometrically distorted image of an equirectangular projection can be improved by automatically acquiring a feature point matching pair associated with a matching target building between the street view data 120. In addition, by using the 3D model 110 containing 3D geometric information expressed in absolute positions, it is possible to determine the same matching target building among numerous buildings with high appearance similarity and automatically obtain matching pairs of feature points for the building. Therefore, the cost and effort of obtaining matching pairs of feature points for each of a plurality of buildings included in the street view data 120 can be reduced. Automatically acquired feature point matching pairs between matching target buildings are used to determine the relative posture between street view data (120) captured at different nodes, providing 3D geometric information, high-accuracy location information and direction information, and high-quality It can be used for various services that utilize all texture information.

Figure 2 is a schematic diagram showing a configuration in which the information processing system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals 210_1, 210_2, and 210_3. As shown, a plurality of user terminals 210_1, 210_2, and 210_3 may be connected to an information processing system 230 capable of providing a map information service through a network 220. Here, the plurality of user terminals 210_1, 210_2, and 210_3 may include terminals of users receiving a map information service. Additionally, the plurality of user terminals 210_1, 210_2, and 210_3 may be cars that capture street view images from nodes. In one embodiment, the information processing system 230 includes one or more server devices and/or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to providing map information services, etc. Alternatively, it may include one or more distributed computing devices and/or distributed databases based on cloud computing services.

The map information service provided by the information processing system 230 may be provided to the user through an application or web browser installed on each of the plurality of user terminals 210_1, 210_2, and 210_3. For example, the information processing system 230 may provide information corresponding to a street view image request, an image-based location recognition request, etc. received from the user terminals 210_1, 210_2, and 210_3 through an application or perform corresponding processing. You can.

A plurality of user terminals 210_1, 210_2, and 210_3 may communicate with the information processing system 230 through the network 220. The network 220 may be configured to enable communication between a plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of wireless networks such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include communication methods utilizing communication networks that the network 220 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.) as well as user terminals (210_1, 210_2, 210_3). ) may also include short-range wireless communication between

In Figure 2, the mobile phone terminal (210_1), tablet terminal (210_2), and PC terminal (210_3) are shown as examples of user terminals, but they are not limited thereto, and the user terminals (210_1, 210_2, 210_3) use wired and/or wireless communication. This may be any computing device capable of installing and executing an application or a web browser. For example, user terminals include AI speakers, smartphones, mobile phones, navigation, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, It may include wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, set-top boxes, etc. In addition, in Figure 2, three user terminals (210_1, 210_2, 210_3) are shown as communicating with the information processing system 230 through the network 220, but this is not limited to this, and a different number of user terminals are connected to the network ( It may be configured to communicate with the information processing system 230 through 220).

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing an application or a web browser and capable of wired/wireless communication, for example, the mobile phone terminal 210_1, tablet terminal 210_2 of FIG. 2, It may include a PC terminal (210_3), etc. As shown, the user terminal 210 may include a memory 312, a processor 314, a communication module 316, and an input/output interface 318. Similarly, information processing system 230 may include memory 332, processor 334, communication module 336, and input/output interface 338. As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using respective communication modules 316 and 336. It can be. Additionally, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or to output information and/or data generated from the user terminal 210.

Memories 312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the memories 312 and 332 are non-permanent mass storage devices such as read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It can be included. As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the user terminal 210 or the information processing system 230 as a separate persistent storage device that is distinct from memory. Additionally, the memories 312 and 332 may store an operating system and at least one program code (eg, code for an application installed and running on the user terminal 210).

These software components may be loaded from a computer-readable recording medium separate from the memories 312 and 332. This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, disk, tape, DVD/CD- It may include computer-readable recording media such as ROM drives and memory cards. As another example, software components may be loaded into the memories 312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into memory 312, 332 based on a computer program installed by files provided over the network 220 by developers or a file distribution system that distributes installation files for applications. It can be.

The processors 314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processors 314 and 334 by memories 312 and 332 or communication modules 316 and 336. For example, processors 314 and 334 may be configured to execute received instructions according to program codes stored in recording devices such as memories 312 and 332.

The communication modules 316 and 336 may provide a configuration or function for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and may provide a configuration or function for the user terminal 210 and/or information processing. The system 230 may provide a configuration or function for communicating with other user terminals or other systems (for example, a separate cloud system, etc.). For example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as the memory 312 (e.g., data associated with a request for a street view image taken on the ground) etc.) may be transmitted to the information processing system 230 through the network 220 under the control of the communication module 316. Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 is transmitted through the communication module 316 of the user terminal 210 through the communication module 336 and the network 220. It may be received by the user terminal 210. For example, the user terminal 210 may receive data related to a street view image for a specific area from the information processing system 230.

The input/output interface 318 may be a means for interfacing with the input/output device 320. As an example, input devices may include devices such as cameras, keyboards, microphones, mice, etc., including audio sensors and/or image sensors, and output devices may include devices such as displays, speakers, haptic feedback devices, etc. You can. As another example, the input/output interface 318 may be a means for interfacing with a device that has components or functions for performing input and output, such as a touch screen, integrated into one. For example, the processor 314 of the user terminal 210 uses information and/or data provided by the information processing system 230 or another user terminal when processing instructions of a computer program loaded in the memory 312. A service screen, etc. configured as such may be displayed on the display through the input/output interface 318. In FIG. 3 , the input/output device 320 is shown not to be included in the user terminal 210, but the present invention is not limited to this and may be configured as a single device with the user terminal 210. Additionally, the input/output interface 338 of the information processing system 230 is connected to the information processing system 230 or means for interfacing with a device (not shown) for input or output that the information processing system 230 may include. It can be. In FIG. 3 , the input/output interfaces 318 and 338 are shown as elements configured separately from the processors 314 and 334, but the present invention is not limited thereto, and the input/output interfaces 318 and 338 may be configured to be included in the processors 314 and 334. there is.

The user terminal 210 and information processing system 230 may include more components than those in FIG. 3 . However, there is no need to clearly show most prior art components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the input/output devices 320 described above. Additionally, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, if the user terminal 210 is a smartphone, it may include components generally included in a smartphone, such as an acceleration sensor, a gyro sensor, an image sensor, a proximity sensor, a touch sensor, Various components such as an illuminance sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and a vibrator for vibration may be implemented to be further included in the user terminal 210. According to one embodiment, the processor 314 of the user terminal 210 may be configured to operate an application that provides a map information service. At this time, code associated with the corresponding application and/or program may be loaded into the memory 312 of the user terminal 210.

While a program for an application providing a map information service is running, the processor 314 uses input devices such as a touch screen, a keyboard, a camera including an audio sensor and/or an image sensor, and a microphone connected to the input/output interface 318. It is possible to receive text, images, videos, voices, and/or actions input or selected through, and store the received text, images, videos, voices, and/or actions in the memory 312 or use the communication module 316 and It can be provided to the information processing system 230 through the network 220. For example, the processor 314 may receive a user's input requesting a street view image for a specific area and provide the input to the information processing system 230 through the communication module 316 and the network 220.

The processor 314 of the user terminal 210 manages, processes, and/or stores information and/or data received from the input/output device 320, other user terminals, the information processing system 230, and/or a plurality of external systems. It can be configured to do so. Information and/or data processed by processor 314 may be provided to information processing system 230 via communication module 316 and network 220. The processor 314 of the user terminal 210 may transmit information and/or data to the input/output device 320 through the input/output interface 318 and output the information. For example, the processor 314 may display the received information and/or data on the screen of the user terminal.

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals 210 and/or a plurality of external systems. Information and/or data processed by the processor 334 may be provided to the user terminal 210 through the communication module 336 and the network 220.

FIG. 4 is a diagram illustrating an example of determining a matching candidate area among a plurality of buildings included in a street view image captured from the ground according to an embodiment of the present disclosure. The information processing system may determine a matching candidate area for acquiring a plurality of feature point matching pairs based on a street view image captured on the ground. Here, the street view image may be a 360-degree panoramic image captured using a vehicle equipped with at least one camera. That is, the street view image may be a panoramic image created using equirectangular projection. Additionally, street view images may include low-precision GPS location information acquired when shooting.

According to one embodiment, the information processing system may determine a matching candidate area associated with a matching target building among a plurality of buildings included in the street view image, based on a street view image for a specific area and a 3D model for a specific area. there is. Here, the 3D model may include 3D building models for a plurality of buildings.

As a specific example, a matching candidate area may be determined through the first state 410 and the second state 420. The first state 410 represents an example of a result of projecting a 3D building model for a plurality of buildings onto a street view image. According to one embodiment, the information processing system may project a 3D building model for a plurality of buildings onto the street view image using the location information and direction information of the street view image. For example, as shown, a plurality of 3D absolute coordinate position information associated with a 3D building model for each of a plurality of buildings (e.g., 3D absolute coordinate position information of a vertex of the 3D building model) ) can be converted into a plurality of two-dimensional points on the spherical coordinate system of the street view image and projected onto the street view image.

The second state 420 represents an example of a result in which an area associated with each of a plurality of buildings in the street view image is determined based on the projection result. According to one embodiment, the information processing system may determine an area associated with each of a plurality of buildings in the street view image based on the projection result. For example, the information processing system may determine a minimum image area including a plurality of two-dimensional points for each of a plurality of buildings as an area associated with the building. Here, the minimum image area can be determined using the Convex Hull algorithm, etc.

Then, the information processing system may determine a matching candidate area associated with the matching target building, based on the area associated with each of the plurality of buildings, by excluding buildings that are obscured by other buildings. To determine which building is obscured by another building, 3D absolute coordinate location information included in the 3D building model can be used. For example, determine the distance from the location where the street view image was taken to each building, and based on that distance, match the distant building if the area of the building located further away is obscured by the area of the building located closer. It can be excluded from the target building.

For example, as shown, based on the projection result, a minimum image area containing a two-dimensional point for each of a plurality of buildings may be determined as an area associated with the building. That is, the first building area 422, the second building area 424, the third building area 426, and the fourth building area 428 may be determined based on two-dimensional points associated with different buildings. The information processing system then controls the first building area 422, the second building area 424, and the third building area that are not obscured by other buildings, except for the fourth building area 428 that is obscured by the other building. Area 426 may be determined as the matching target building.

According to one embodiment, the information processing system may perform feature matching based on an area associated with a matching target building that is commonly included in a plurality of street view images. Specifically, the information processing system may use a 3D building model for the matching target building included in the first street view image to determine the area associated with the matching target building in the first street view image as the first matching candidate area. Additionally, the information processing system may use a 3D building model for a specific building included in the second street view image to determine an area associated with a specific building in the second street view image as the second matching candidate area. Then, in response to determining that the first matching candidate area and the second matching candidate area are associated with the matching target building, the information processing system performs feature matching based on the first matching candidate area and the second matching candidate area. can do.

Figure 4 shows that the first to fourth building areas 422 to 428 are determined among a plurality of buildings in the street view image, but this is for convenience of explanation, and the information processing system is included in the street view image. Each building area can be determined for all of the plurality of buildings, and among them, the building that is not obscured by other buildings can be determined as the matching target building.

Figure 5 shows a process of performing feature matching based on the first matching candidate area 512 in the first street view image and the second matching candidate area 522 in the second street view image according to an embodiment of the present disclosure. This is a drawing showing an example. According to one embodiment, the information processing system uses the first matching candidate area 512 and the second matching candidate area to perform feature matching based on the first matching candidate area 512 and the second matching candidate area 522. Each area 522 can be converted into a flat image. For example, as shown, the information processing system may convert the first matching candidate area 512 into a first planar image 514 using a perspective projection method. Specifically, the information processing system may determine the direction and size of a specific matching target building in the spherical coordinate system of the first street view image. And, the information processing system may convert the first matching candidate area 512 into a first planar image 514 based on the determined direction and size of the first building. Similarly, the information processing system may convert the second matching candidate area 522 into a second planar image 524 using a perspective projection method. The information processing system may then perform feature matching between the first planar image 514 and the second planar image 524.

As such, according to an embodiment of the present disclosure, in order to obtain a matching pair of feature points associated with a matching target building between street view data, a geometric distortion image of an equirectangular projection including a matching target building is converted into a flat image, and then , the quality of feature matching for street view images can be improved by performing feature matching.

FIG. 6 is a diagram illustrating an example of a plurality of feature points detected from a planar image 600 of a matching target building according to an embodiment of the present disclosure. Here, the planar image 600 for the matching target building may represent an undistorted planar image generated based on a matching candidate area associated with the matching target building in the street view image.

According to one embodiment, the information processing system may detect a plurality of local feature points for generating a feature point matching pair associated with the matching target building from the planar image 600 of the matching target building. For example, the information processing system may detect a plurality of local feature points from the planar image 600 and then determine the feature point associated with the matching target building. Additionally, the information processing system can extract a visual descriptor of each feature point. Multiple feature points can be extracted using SIFT (Scale-Invariant Feature Transform), SuperPoint, and R2D2 (Repeatable and Reliable Detector and Descriptor) techniques, but are not limited to this.

FIG. 7 is a diagram illustrating an example of a matching result between a first planar image 710 and a second planar image 720 according to an embodiment of the present disclosure. According to one embodiment, the information processing system may extract a plurality of point correspondence pairs for matching target buildings between the first planar image 710 and the second planar image 720 and store them as matching results. Here, the first planar image 710 and the second planar image 720 are images of the matching target building created based on the first street view image and the second street view image captured at different nodes in a specific area. You can.

As shown, the information processing system includes a first set of feature points extracted from the first planar image 710 for the matching target building and a second set of feature points extracted from the second planar image 720 for the matching target building. A plurality of point correspondence pairs can be extracted and stored as a feature matching result for the matching target building. Here, the first set of feature points and the second set of feature points may be extracted using the same or the same type of feature extractor. The feature extractor is a visual feature extraction neural network model, and the visual feature descriptor of the corresponding pixel between the plurality of planar images (710, 720) generated by converting the street view image is similar. It may be a model that has been trained to extract the data correctly. Additionally, the method of performing feature matching is not limited to a specific method, and any feature matching method (NN Matching, SuperPoint/Glue, deep learning-based feature matching method such as R2D2, etc.) can be used.

FIG. 8 is a diagram illustrating an example of a feature point matching result 820 between street view images acquired based on a matching result 810 for each of a plurality of matching target buildings according to an embodiment of the present disclosure. The information processing system may obtain matching results 810 for each of a plurality of matching target buildings included in the street view image using the method described in FIGS. 4 to 7. For example, the matching result 810 for each of a plurality of matching target buildings may be a matching result between the first planar image 812_1 and the second planar image 812_2 for the first building, and the third planar image 812_2 for the second building. It may include a matching result between the flat image 814_1 and the fourth flat image 814_2, and a matching result between the fifth flat image 816_1 and the sixth flat image 816_2 for the third building. Here, the first building, second building, and third building may represent different buildings included in the street view image associated with each planar image.

According to one embodiment, the information processing system may acquire a feature point matching result 820 between street view images in a spherical coordinate system based on a matching result 810 for each of a plurality of matching target buildings in a planar coordinate system. Specifically, the information processing system can convert two-dimensional coordinate position information in a planar image of a plurality of feature point matching pairs into two-dimensional position information in a street view image using inversion of the perspective projection method. For example, the information processing system uses two-dimensional coordinate position information in the first planar image 812_1 and two-dimensional coordinate position information in the second planar image 812_2 of the plurality of feature point matching pairs for the first building, respectively. It can be converted into 2D location information in the first street view image 822 and 2D location information in the second street view image 824. Similarly, the information processing system includes two-dimensional coordinate location information in the planar images 814_1 and 814_2 of the plurality of feature point matching pairs for the second building and the planar images 816_1 and the plurality of feature point matching pairs for the third building. The 2D coordinate location information in 816_2) can be converted into 2D location information in the first street view image 822 and 2D location information in the second street view image 824.

Through this configuration, matching pairs of feature points between street view images are quickly and accurately acquired automatically for multiple buildings (e.g., first building, second building, third building, etc.) included in different street view images. can do.

FIG. 9 is a diagram illustrating an example of selecting a pair of street view images that commonly include the same building based on the street view image map 900. The street view image map 900 may include relationship information between a plurality of street view images in a specific area. For example, referring to FIG. 9 , circles or dots may represent nodes where each of a plurality of street view images is captured. Additionally, edges or lines connecting each node may indicate relationships between street view images that commonly include buildings included in the street view images.

According to one embodiment, the information processing system may select a pair of street view images that commonly include the same building in a plurality of street view images. For example, the information processing system may receive a plurality of street view image maps 900 captured from a plurality of nodes within a specific area. Then, the information processing system uses the three-dimensional model to display a first street view image captured at the first node 910 and a second street view image captured at the second node 920 on the plurality of street view image maps 900. It may be determined that the same building is commonly included in the street view image and the third street view image captured by the third node 930. In this case, the information processing system includes a first street view image captured at the first node 910, a second street view image captured at the second node 920, and a third street view image captured at the third node 930. A combination of two street view images among the images can be selected as a pair of street view images to perform feature matching.

Figure 9 shows an example of selecting a street view image pair based on three street view images taken at three nodes, but the present invention is not limited to this, and the street view image pair is two street view images among a plurality of street view images. It can be made up of a combination of .

FIG. 10 is a flowchart illustrating an example of a method 1000 for acquiring feature point matching pairs between street view images using a 3D model according to an embodiment of the present disclosure. Method 1000 may be initiated by a processor (e.g., at least one processor of an information processing system) receiving a three-dimensional model for a specific region containing three-dimensional geometric information expressed in absolute positions (S1010). . Here, the 3D model may include multiple 3D building models for multiple buildings.

Additionally, the processor may receive a plurality of street view images taken from a plurality of nodes within a specific area (S1020). Here, the street view image may be a panoramic image created using equirectangular projection. For example, the plurality of nodes includes a first node and a second node, and the processor generates a first street view image captured at a first node within a specific area and a second street view image captured at a second node within a specific area. can receive. Here, the first street view image and the second street view image may include the first building. Additionally, the 3D model may include a 3D building model for the first building.

In one embodiment, the processor may determine that the first street view image and the second street view image include the first building using a 3D model (S1030). In this case, the processor may select the first street view image and the second street view image among the plurality of street view images as a pair of street view images to perform feature matching (S1040). Afterwards, the processor may acquire a plurality of feature point matching pairs based on the first street view image, the second street view image, and the 3D model (S1050). For example, multiple feature point matching pairs can be acquired using NN Matching or SuperGlue techniques.

In one embodiment, the processor determines an area associated with the first building in the first street view image as the first matching candidate area using a 3D building model for the first building to obtain a plurality of feature point matching pairs. You can. Similarly, the processor may use the 3D building model for the first building to determine an area associated with the first building in the second street view image as the second matching candidate area. The processor may then perform feature matching based on the first matching candidate area and the second matching candidate area in response to determining that the first matching candidate area and the second matching candidate area are associated with the first building. .

In one embodiment, the step of determining an area associated with the first building in the first street view image as the first matching candidate area includes using location information and direction information associated with the first street view image, It may include projecting the 3D building model onto the first street view image and determining an area associated with the first building in the first street view image based on the projection result. Here, the step of projecting the 3D building model for the first building onto the first street view image involves projecting the 3D building model for the first building using location information and direction information associated with the first street view image. It may include converting a plurality of 3D absolute coordinate location information into a plurality of 2D points on a spherical coordinate system of the first street view image. Additionally, determining the area associated with the first building in the first street view image based on the projection result may include determining a minimum image area including a plurality of two-dimensional points as the area associated with the first building. You can. Here, the minimum image area can be determined using the Convex Hull algorithm.

In one embodiment, the 3D model may further include a 3D building model for a second building. In this case, the step of acquiring a plurality of feature point matching pairs includes determining an area associated with the second building in the first street view image using a 3D building model for the second building and an area associated with the first building. It may include excluding an area obscured by an area associated with the second building from the first matching candidate area. In this case, the distance from the first node to the first building may be greater than the distance from the first node to the second building.

In one embodiment, performing feature matching based on the first matching candidate area and the second matching candidate area includes converting the first matching candidate area into a first planar image using a perspective projection method, the step of converting the first matching candidate area into a first planar image using a perspective projection method. It may include converting the second matching candidate area into a second planar image using a method and performing feature matching between the first planar image and the second planar image. Here, the step of converting the first matching candidate area into a first planar image includes determining the direction and size of the first building in the spherical coordinate system of the first street view image, and based on the determined direction and size of the first building , It may include converting the first matching candidate area into a first planar image.

In one embodiment, performing feature matching between the first planar image and the second planar image includes extracting a first set of feature points based on the first planar image, and extracting a first set of feature points based on the second planar image. It may include extracting feature points of the set, performing feature matching between feature points of the first set and feature points of the second set, and acquiring a plurality of feature point matching pairs based on the feature matching results. You can. Here, the first set of feature points and the second set of feature points may be extracted using the same type of feature extractor. For example, the first set of feature points and the second set of feature points may be extracted using SIFT (Scale-Invariant Feature Transform), SuperPoint, or R2D2 (Repeatable and Reliable Detector and Descriptor) techniques.

In one embodiment, the step of acquiring a plurality of feature point matching pairs based on the feature matching result includes two-dimensional coordinate position information in the first plane image of the plurality of feature point matching pairs using inversion of a perspective projection method, and a second pair of feature point matching pairs. It may include converting 2D coordinate location information in the two-plane image into 2D location information in the first street view image and 2D location information in the second street view image, respectively.

The flowchart of FIG. 10 and the above description are only examples, and the scope of the present disclosure is not limited thereto. For example, at least one step may be added/changed/deleted, or the order of each step may be changed.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by a person skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

110: 3D model
112: 3D building model 114: Digital elevation model 116: Orthoimagery
120: Street view data
130: Map matching 132: Map matching point/line
150: Feature matching 152: Feature point correspondence set
160: Estimation of location information and direction information
162: Absolute coordinate position information and direction information

Claims (19)

In a method of acquiring feature point matching pairs between street view images using a 3D model, performed by at least one processor,
Receiving a 3D model for a specific area including 3D geometric information expressed in absolute position;
Receiving a first street view image captured at a first node within the specific area, wherein the first street view image includes a first building;
Receiving a second street view image captured at a second node within the specific area, wherein the second street view image includes the first building; and
Obtaining a plurality of feature point matching pairs based on the first street view image, the second street view image, and the 3D model.
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to paragraph 1,
The first street view image and the second street view image are panoramic images generated by equirectangular projection. A method of acquiring matching pairs of feature points between street view images using a three-dimensional model. According to paragraph 1,
The three-dimensional model includes a three-dimensional building model for the first building,
The step of acquiring the plurality of feature point matching pairs is,
Using a 3D building model for the first building, determining an area associated with the first building in the first street view image as a first matching candidate area;
Using a 3D building model for the first building, determining an area associated with the first building in the second street view image as a second matching candidate area; and
In response to determining that the first matching candidate area and the second matching candidate area are associated with the first building, performing feature matching based on the first matching candidate area and the second matching candidate area.
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to paragraph 3,
The step of determining an area associated with the first building in the first street view image as a first matching candidate area,
Projecting a 3D building model for the first building onto the first street view image using location information and direction information associated with the first street view image; and
determining an area associated with the first building in the first street view image based on the projection result.
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to clause 4,
The projection step is,
Using the location information and direction information associated with the first street view image, a plurality of 3D absolute coordinate location information associated with the 3D building model for the first building is divided into a plurality of 3D absolute coordinate location information on the spherical coordinate system of the first street view image. Steps to convert to 2D points
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to clause 5,
Determining an area associated with the first building in the first street view image based on the projection result includes:
Determining a minimum image area including the plurality of two-dimensional points as an area associated with the first building
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to clause 6,
A method of acquiring matching pairs of feature points between street view images using a 3D model, wherein the minimum image area is determined using the Convex Hull algorithm. According to paragraph 3,
The three-dimensional model further includes a three-dimensional building model for the second building,
The step of acquiring the plurality of feature point matching pairs is,
determining an area associated with the second building in the first street view image using a 3D building model for the second building; and
Excluding an area obscured by an area associated with the second building among areas associated with the first building from the first matching candidate area.
A method of acquiring matching pairs of feature points between street view images using a 3D model, further comprising: According to clause 8,
A method of acquiring matching pairs of feature points between street view images using a 3D model, wherein the distance from the first node to the first building is greater than the distance from the first node to the second building. According to paragraph 3,
The step of performing the feature matching is,
converting the first matching candidate area into a first planar image using a perspective projection method;
converting the second matching candidate area into a second planar image using a perspective projection method; and
performing feature matching between the first planar image and the second planar image.
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to clause 10,
The step of converting to the first flat image is,
determining the direction and size of the first building in a spherical coordinate system of the first street view image; and
Converting the first matching candidate area into the first planar image based on the determined direction and size of the first building.
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to clause 10,
The step of performing feature matching between the first planar image and the second planar image includes:
extracting a first set of feature points based on the first planar image;
extracting a second set of feature points based on the second planar image;
performing feature matching between the first set of feature points and the second set of feature points; and
acquiring the plurality of feature point matching pairs based on the feature matching results;
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to clause 12,
The step of acquiring the plurality of feature point matching pairs based on the feature matching result,
Using inversion of the perspective projection method, the two-dimensional coordinate position information of the plurality of feature point matching pairs in the first plane image and the two-dimensional coordinate position information in the second plane image are respectively converted into the first street view image. Converting the 2D location information in and the 2D location information in the second street view image.
Method for acquiring feature point matching pairs between street view images using a 3D model, including. According to clause 12,
A method of acquiring a matching pair of feature points between street view images using a 3D model, wherein the feature points of the first set and the feature points of the second set are extracted using the same type of feature extractor. According to clause 12,
The first set of feature points and the second set of feature points are extracted using SIFT (Scale-Invariant Feature Transform), SuperPoint, or R2D2 (Repeatable and Reliable Detector and Descriptor) techniques, between street view images using a 3D model. Method for acquiring feature point matching pairs. According to clause 12,
A method of acquiring feature point matching pairs between street view images using a 3D model, wherein the plurality of feature point matching pairs are acquired using NN Matching or SuperGlue techniques. According to paragraph 1,
Receiving a plurality of street view images taken from a plurality of nodes within the specific area, wherein the plurality of nodes include the first node and the second node, and the plurality of street view images are the first street view image. including video and the second street view video;
determining that the first building is included in the first street view image and the second street view image using the 3D model; and
Selecting the first street view image and the second street view image among the plurality of street view images as a pair of street view images to perform feature matching.
A method of acquiring matching pairs of feature points between street view images using a 3D model, further comprising: A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1 to 17 on a computer. As an information processing system,
communication module;
Memory; and
At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory
Including,
The at least one program is,
Receive a 3D model for a specific area containing 3D geometric information expressed in absolute position,
Receiving a first street view image taken at a first node within the specific area, wherein the first street view image includes a first building,
Receiving a second street view image taken at a second node within the specific area, wherein the second street view image includes the first building,
An information processing system comprising instructions for acquiring a plurality of feature point matching pairs based on the first street view image, the second street view image, and the three-dimensional model.

Images