WO2024106833A1

WO2024106833A1 - Method and system for automatically acquiring building control point

Info

Publication number: WO2024106833A1
Application number: PCT/KR2023/017705
Authority: WO
Inventors: 조원준
Original assignee: 네이버랩스 주식회사
Priority date: 2022-11-16
Filing date: 2023-11-06
Publication date: 2024-05-23
Also published as: KR20240071697A

Abstract

The present disclosure relates to a method, performed by at least one processor, for automatically acquiring a building control point. The method for automatically acquiring a building control point comprises the steps of: receiving a street view image captured on the ground, the street view image comprising a plurality of buildings including a matching target building; receiving an aerially captured image related to the street view image, the aerially captured image comprising the matching target building; and acquiring, on the basis of the street view image and the aerially captured image, a plurality of building control points related to the matching target building.

Description

Building control point automatic acquisition method and system

This disclosure relates to a method and system for automatically acquiring building control points. Specifically, by performing feature matching using street view images and aerial images captured on the ground, a plurality of building control points associated with a matching target building are acquired. It relates to methods and systems.

Ground Control Point refers to the ground reference point used to obtain the coordinate conversion equation between the image coordinate system and the map coordinate system, that is, absolute coordinate location information. In order to acquire ground control points, they can be acquired by measuring the location information of markers installed on the ground using equipment such as a high-precision global positioning system (GPS).

Meanwhile, as information technology develops, map information services are commercialized, and street view images are provided as an area of map information services. For example, a map information service provider can obtain images of a real space using a moving object on the ground and then provide the images taken at a specific point on an electronic map as a street view image.

However, street view images do not contain absolute coordinate location information, and in order to acquire ground control points for street view images taken using moving objects on the ground, a large number of markers must be installed over a wide range, which requires many There is a problem that it requires money and effort. Additionally, in complex environments in urban areas, ground control points may not be observed due to obstacles, etc., so when only ground control points are used, the reliability of vehicle pose estimation, etc. may not be sufficient.

The present disclosure provides a method for solving the above problems, a computer-readable non-transitory recording medium on which instructions are recorded, and a device (system).

The present disclosure may be implemented in various ways, including a method, a device (system), or a computer-readable non-transitory recording medium recording instructions.

According to an embodiment of the present disclosure, a method of automatically acquiring a building control point, performed by at least one processor, includes receiving a street view image taken from the ground - a plurality of buildings including a matching target building in the street view image. This includes - receiving an aerial image associated with the street view image - the aerial image includes a matching target building - and based on the street view image and the aerial image, a plurality of building control points associated with the matching target building. Includes acquisition steps.

A computer-readable non-transitory recording medium recording instructions for executing a method according to an embodiment of the present disclosure on a computer is provided.

According to an embodiment of the present disclosure, an information processing system includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, and at least one The program receives a street view image taken from the ground - the street view image includes a plurality of buildings including the building to be matched - and receives an aerial image associated with the street view image - the aerial view image includes a building to be matched This includes commands for acquiring a plurality of building control points associated with a matching target building based on street view images and aerial images.

According to an embodiment of the present disclosure, a plurality of high-quality building control points associated with a matching target building can be automatically acquired through feature matching between an aerial image and a street view image including absolute coordinate information.

According to an embodiment of the present disclosure, costs can be reduced by omitting the step of installing markers on the ground, and the building control point of consistent quality is achieved by omitting the step of manually measuring the location information of markers installed on the ground. can be acquired.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clear to a person skilled in the art (referred to as “a person skilled in the art”) in the technical field to which this disclosure pertains from the description of the claims. It will be understandable.

Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals indicate like elements, but are not limited thereto.

1 is a diagram illustrating an example of a method for matching a 3D model and street view data according to an embodiment of the present disclosure.

Figure 2 is a schematic diagram showing a configuration in which an information processing system according to an embodiment of the present disclosure is connected to enable communication with a plurality of user terminals.

Figure 3 is a block diagram showing the internal configuration of a user terminal and an information processing system according to an embodiment of the present disclosure.

FIG. 4 is a diagram illustrating an example of determining a matching target building among a plurality of buildings included in a street view image captured from the ground according to an embodiment of the present disclosure.

Figure 5 is a diagram illustrating an example of acquiring a building control point by performing feature matching between an aerial capture image and a street view image according to an embodiment of the present disclosure.

Figure 6 is a block diagram showing a specific method of acquiring a building control point by performing feature matching between a 3D model and street view data according to an embodiment of the present disclosure.

FIG. 7 is a diagram illustrating an example of an image in which a viewpoint composite image associated with an aerial shot image is projected onto a planar image associated with a street view image according to an embodiment of the present disclosure.

FIG. 8 is a diagram illustrating an example of a matching result between a viewpoint composite image and a planar image according to an embodiment of the present disclosure.

FIG. 9 is a diagram illustrating an example of a plurality of building control points matched between a street view image and an aerial image for a specific matching target building according to an embodiment of the present disclosure.

Figure 10 is a flowchart illustrating an example of a method for automatically acquiring a building control point according to an embodiment of the present disclosure.

Hereinafter, specific details for implementing the present disclosure will be described in detail with reference to the attached drawings. However, in the following description, detailed descriptions of well-known functions or configurations will be omitted if there is a risk of unnecessarily obscuring the gist of the present disclosure.

In the accompanying drawings, identical or corresponding components are given the same reference numerals. Additionally, in the description of the following embodiments, overlapping descriptions of identical or corresponding components may be omitted. However, even if descriptions of components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments and methods for achieving them will become clear by referring to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete and that the present disclosure does not convey the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification are general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedent of a technician working in the related field, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Accordingly, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of the present disclosure, rather than simply the name of the term.

In this specification, singular expressions include plural expressions, unless the context clearly specifies the singular. Additionally, plural expressions include singular expressions, unless the context clearly specifies plural expressions. When it is said that a certain part includes a certain element throughout the specification, this does not mean excluding other elements, but may further include other elements, unless specifically stated to the contrary.

Additionally, the term 'module' or 'unit' used in the specification refers to a software or hardware component, and the 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium and may be configured to run on one or more processors. Thus, as an example, a 'module' or 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, or variables. Components and 'modules' or 'parts' may be combined into smaller components and 'modules' or 'parts' or further components and 'modules' or 'parts'. Could be further separated.

According to an embodiment of the present disclosure, a 'module' or 'unit' may be implemented with a processor and memory. 'Processor' should be interpreted broadly to include general-purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and graphics processing units (GPUs). In some contexts, 'processor' may refer to an application-specific integrated circuit (ASIC), programmable logic device (PLD), field programmable gate array (FPGA), etc. 'Processor' refers to a combination of processing devices, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such combination of configurations. You may. Additionally, 'memory' should be interpreted broadly to include any electronic component capable of storing electronic information. 'Memory' refers to random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erasable-programmable read-only memory (EPROM), May also refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, etc. A memory is said to be in electronic communication with a processor if the processor can read information from and/or write information to the memory. The memory integrated into the processor is in electronic communication with the processor.

In the present disclosure, 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In this disclosure, 'display' may refer to any display device associated with a computing device, e.g., any display device capable of displaying any information/data controlled by or provided by the computing device. can refer to.

In the present disclosure, 'each of a plurality of A' or 'each of a plurality of A' may refer to each of all components included in a plurality of A, or may refer to each of some components included in a plurality of A. .

In the present disclosure, 'street view data' may refer to data including road view data including images captured on the roadway and location information, as well as walk view data including images captured on the sidewalk and location information. . In addition, 'street view data' may further include images and location information taken at random points outdoors (or indoors facing the outdoors), as well as roadways and sidewalks.

FIG. 1 is a diagram illustrating an example of a method of matching a 3D model 110 and street view data 120 according to an embodiment of the present disclosure. The information processing system may acquire/receive the 3D model 110 and street view data 120 for a specific area.

The 3D model 110 may include 3D geometric information expressed in absolute coordinate positions and texture information corresponding thereto. Here, the location information included in the 3D model 110 may be information of higher accuracy than the location information included in the street view data 120. Additionally, the texture information included in the 3D model 110 may be of lower quality (eg, lower resolution) than the texture information included in the street view data 120. According to one embodiment, 3D geometric information expressed as an absolute coordinate position may be generated based on an aerial photograph taken of a specific area from above the specific area.

The 3D model (110) for a specific area includes a 3D building model (112), a digital elevation model (DEM) (114), a true ortho image (116) for a specific area, and a digital surface. It may include a digital surface model (DSM), road layout, road DEM, etc. As a specific example, the 3D model 110 for a specific area includes a digital surface model (DSM) containing geometric information about the ground of the specific area and an orthoimage 116 for the specific area corresponding thereto. It may be a model created based on, but is not limited to, this. In one embodiment, a precise orthoimage 116 of a specific area may be generated based on a plurality of aerial photos and the absolute coordinate location information and direction information of each aerial photo.

The street view data 120 may include a plurality of street view images captured at a plurality of nodes within a specific area and absolute coordinate location information for each of the plurality of street view images. Here, the location information included in the street view data 120 may be information of lower accuracy than the location information included in the 3D model 110, and the texture information included in the street view image is included in the 3D model 110. It may be information of higher quality (e.g., higher resolution) than the included texture information. For example, the location information included in the street view data 120 may be location information obtained using a GPS device when a node captures a street view image. Location information obtained using a vehicle's GPS equipment may have an error of about 5 to 10 meters. Additionally, street view data may include direction information (i.e., image shooting direction information) for each of a plurality of street view images.

The information processing system may perform map matching 130 between the 3D model 110 and street view data 120. Specifically, the information processing system may perform feature matching between texture information included in the 3D model 110 and a plurality of street view images included in the street view data 120. To perform map matching 130, the information processing system may convert at least some of the plurality of street view images included in the street view data 120 into a top view image. As a result of map matching 130, a plurality of map matching points/map matching lines 132 can be extracted.

The map matching point may represent a corresponding pair of a point in the street view image and a point in the 3D model 110. The type of map matching point may vary depending on the type of 3D model 110 used for map matching 130, the location of the point, etc. For example, map matching points are Ground Control Points (GCP), which are point correspondence pairs on the ground within a specific area, and Building Control Points (BCP), which are point correspondence pairs on buildings within a specific area. Alternatively, it may include at least one of structure control points, which are point correspondence pairs in structures within a specific area. Map matching points can be extracted not only from the ground, buildings, and structures described above, but also from street view images and arbitrary areas of the 3D model 110. In the present disclosure, a method of extracting building control points, which are point correspondence pairs in buildings within a specific area, using street view data 120 and the 3D model 110, will be described in detail later with reference to FIGS. 4 to 10. .

The map matching line may represent a corresponding pair of one line of the street view image and one line of the 3D model 110. The type of map matching line may vary depending on the type of 3D model 110 used for map matching 130, the location of the line, etc. For example, map matching lines include Ground Control Line (GCL), which is a corresponding pair of lines on the ground within a specific area, and Building Control Line (BCL), which is a corresponding pair of lines on buildings within a specific area. , it may include at least one of a structure control line, which is a corresponding pair of lines in a structure within a specific area, or a lane control line, which is a corresponding pair of lines in a lane within a specific area. Map matching lines can be extracted from the ground, buildings, structures, and lanes described above, as well as street view images and arbitrary areas of the 3D model 110.

Additionally, the information processing system may perform feature matching 150 between a plurality of street view images to extract a plurality of feature point correspondence sets 152. According to one embodiment, for robust feature matching, feature matching 150 between a plurality of street view images may be performed using at least a portion of the 3D model 110. For example, feature matching 150 between street view images can be performed using the 3D building model 112 included in the 3D model 110.

Then, the information processing system provides absolute coordinate position information and Direction information can be estimated (160). For example, the processor may estimate absolute coordinate position information and direction information for a plurality of street view images using a bundle adjustment technique (160). According to one embodiment, the estimated absolute coordinate position information and direction information 162 is information in an absolute coordinate system representing the 3D model 110, and may be a parameter of 6 degrees of freedom (DoF). The absolute coordinate location information and direction information 162 estimated through this process may be data with higher precision than the absolute coordinate location information and direction information included in the street view data 120.

According to an embodiment of the present disclosure, feature matching is performed between the orthoimage (or aerial image) 116 and the street view data 120 included in the 3D model 110 to identify the matching target building within a specific area. By automatically acquiring building control points that are point correspondence pairs, a large number of high-quality building control points can be automatically acquired. In addition, by omitting the steps of installing markers on the ground and manually measuring the location information of markers installed on the ground, costs can be reduced and building control points of consistent quality can be obtained. The automatically acquired building control points are used to match the 3D model 110, which includes 3D geometric information and high-accuracy location and direction information, and the street view data 120, which includes high-quality texture information, 3 It can be used in a variety of services that utilize dimensional geometric information, high-accuracy location and direction information, and high-quality texture information.

Figure 2 is a schematic diagram showing a configuration in which the information processing system 230 according to an embodiment of the present disclosure is connected to communicate with a plurality of user terminals 210_1, 210_2, and 210_3. As shown, a plurality of user terminals 210_1, 210_2, and 210_3 may be connected to an information processing system 230 capable of providing a map information service through a network 220. Here, the plurality of user terminals 210_1, 210_2, and 210_3 may include terminals of users receiving a map information service. Additionally, the plurality of user terminals 210_1, 210_2, and 210_3 may be cars that capture street view images from nodes. In one embodiment, the information processing system 230 includes one or more server devices and/or databases capable of storing, providing, and executing computer-executable programs (e.g., downloadable applications) and data related to providing map information services, etc. Alternatively, it may include one or more distributed computing devices and/or distributed databases based on cloud computing services.

The map information service provided by the information processing system 230 may be provided to the user through an application or web browser installed on each of the plurality of user terminals 210_1, 210_2, and 210_3. For example, the information processing system 230 may provide information corresponding to a street view image request, an image-based location recognition request, etc. received from the user terminals 210_1, 210_2, and 210_3 through an application or perform corresponding processing. You can.

A plurality of user terminals 210_1, 210_2, and 210_3 may communicate with the information processing system 230 through the network 220. The network 220 may be configured to enable communication between a plurality of user terminals 210_1, 210_2, and 210_3 and the information processing system 230. Depending on the installation environment, the network 220 may be, for example, a wired network such as Ethernet, a wired home network (Power Line Communication), a telephone line communication device, and RS-serial communication, a mobile communication network, a wireless LAN (WLAN), It may consist of wireless networks such as Wi-Fi, Bluetooth, and ZigBee, or a combination thereof. The communication method is not limited, and may include communication methods utilizing communication networks that the network 220 may include (e.g., mobile communication networks, wired Internet, wireless Internet, broadcasting networks, satellite networks, etc.) as well as user terminals (210_1, 210_2, 210_3). ) may also include short-range wireless communication between

In Figure 2, the mobile phone terminal (210_1), tablet terminal (210_2), and PC terminal (210_3) are shown as examples of user terminals, but they are not limited thereto, and the user terminals (210_1, 210_2, 210_3) use wired and/or wireless communication. This may be any computing device capable of installing and executing an application or a web browser. For example, user terminals include AI speakers, smartphones, mobile phones, navigation, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, game consoles, It may include wearable devices, IoT (internet of things) devices, VR (virtual reality) devices, AR (augmented reality) devices, set-top boxes, etc. In addition, in Figure 2, three user terminals (210_1, 210_2, 210_3) are shown as communicating with the information processing system 230 through the network 220, but this is not limited to this, and a different number of user terminals are connected to the network ( It may be configured to communicate with the information processing system 230 through 220).

According to one embodiment, the information processing system 230 may receive a street view image captured on the ground and an aerial image associated with the street view image from the user terminals 210_1, 210_2, and 210_3. Then, the information processing system 230 may acquire a plurality of building control points associated with the matching target building based on the received street view image and aerial image. Thereafter, the information processing system 230 may transmit the acquired plurality of building control points to the user terminals 210_1, 210_2, and 210_3. In addition, the information processing system 230 can transmit various service-related data based on data created by matching a 3D model and street view data using a plurality of building control points to the user terminals 210_1, 210_2, and 210_3. .

Figure 3 is a block diagram showing the internal configuration of the user terminal 210 and the information processing system 230 according to an embodiment of the present disclosure. The user terminal 210 may refer to any computing device capable of executing an application or a web browser and capable of wired/wireless communication, for example, the mobile phone terminal 210_1, tablet terminal 210_2 of FIG. 2, It may include a PC terminal (210_3), etc. As shown, the user terminal 210 may include a memory 312, a processor 314, a communication module 316, and an input/output interface 318. Similarly, information processing system 230 may include memory 332, processor 334, communication module 336, and input/output interface 338. As shown in FIG. 3, the user terminal 210 and the information processing system 230 are configured to communicate information and/or data through the network 220 using

respective communication modules

316 and 336. It can be. Additionally, the input/output device 320 may be configured to input information and/or data to the user terminal 210 through the input/output interface 318 or to output information and/or data generated from the user terminal 210.

Memories

312 and 332 may include any non-transitory computer-readable recording medium. According to one embodiment, the

memories

312 and 332 are non-permanent mass storage devices such as read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. It can be included. As another example, non-perishable mass storage devices such as ROM, SSD, flash memory, disk drive, etc. may be included in the user terminal 210 or the information processing system 230 as a separate persistent storage device that is distinct from memory. Additionally, the

memories

312 and 332 may store an operating system and at least one program code (eg, code for an application installed and running on the user terminal 210).

These software components may be loaded from a computer-readable recording medium separate from the

memories

312 and 332. This separate computer-readable recording medium may include a recording medium directly connectable to the user terminal 210 and the information processing system 230, for example, a floppy drive, disk, tape, DVD/CD- It may include computer-readable recording media such as ROM drives and memory cards. As another example, software components may be loaded into the

memories

312 and 332 through a communication module rather than a computer-readable recording medium. For example, at least one program is loaded into

memory

312, 332 based on a computer program installed by files provided over the network 220 by developers or a file distribution system that distributes installation files for applications. It can be.

The

processors

314 and 334 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the

processors

314 and 334 by

memories

312 and 332 or

communication modules

316 and 336. For example,

processors

314 and 334 may be configured to execute received instructions according to program codes stored in recording devices such as

memories

312 and 332.

The

communication modules

316 and 336 may provide a configuration or function for the user terminal 210 and the information processing system 230 to communicate with each other through the network 220, and may provide a configuration or function for the user terminal 210 and/or information processing. The system 230 may provide a configuration or function for communicating with other user terminals or other systems (for example, a separate cloud system, etc.). For example, a request or data generated by the processor 314 of the user terminal 210 according to a program code stored in a recording device such as the memory 312 (e.g., street view images and aerial images taken on the ground) data associated with the request, etc.) may be transmitted to the information processing system 230 through the network 220 under the control of the communication module 316. Conversely, a control signal or command provided under the control of the processor 334 of the information processing system 230 is transmitted through the communication module 316 of the user terminal 210 through the communication module 336 and the network 220. It may be received by the user terminal 210. For example, the user terminal 210 may receive data related to a street view image and an aerial image for a specific area from the information processing system 230.

The input/output interface 318 may be a means for interfacing with the input/output device 320. As an example, input devices may include devices such as cameras, keyboards, microphones, mice, etc., including audio sensors and/or image sensors, and output devices may include devices such as displays, speakers, haptic feedback devices, etc. You can. As another example, the input/output interface 318 may be a means for interfacing with a device that has components or functions for performing input and output, such as a touch screen, integrated into one. For example, the processor 314 of the user terminal 210 uses information and/or data provided by the information processing system 230 or another user terminal when processing instructions of a computer program loaded in the memory 312. A service screen, etc. constructed by doing so may be displayed on the display through the input/output interface 318. In FIG. 3 , the input/output device 320 is shown not to be included in the user terminal 210, but the present invention is not limited to this and may be configured as a single device with the user terminal 210. Additionally, the input/output interface 338 of the information processing system 230 may be connected to the information processing system 230 or means for interfacing with a device (not shown) for input or output that the information processing system 230 may include. It can be. In FIG. 3, the input/

output interfaces

318 and 338 are shown as elements configured separately from the

processors

314 and 334, but the present invention is not limited thereto, and the input/

output interfaces

318 and 338 may be configured to be included in the

processors

314 and 334. there is.

The user terminal 210 and information processing system 230 may include more components than those in FIG. 3 . However, there is no need to clearly show most prior art components. According to one embodiment, the user terminal 210 may be implemented to include at least some of the input/output devices 320 described above. Additionally, the user terminal 210 may further include other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. For example, if the user terminal 210 is a smartphone, it may include components generally included in a smartphone, such as an acceleration sensor, a gyro sensor, an image sensor, a proximity sensor, a touch sensor, Various components such as an illuminance sensor, a camera module, various physical buttons, buttons using a touch panel, input/output ports, and a vibrator for vibration may be implemented to be further included in the user terminal 210. According to one embodiment, the processor 314 of the user terminal 210 may be configured to operate an application that provides a map information service. At this time, code associated with the corresponding application and/or program may be loaded into the memory 312 of the user terminal 210.

While a program for an application providing a map information service is running, the processor 314 uses input devices such as a touch screen, a keyboard, a camera including an audio sensor and/or an image sensor, and a microphone connected to the input/output interface 318. It is possible to receive text, images, videos, voices, and/or actions input or selected through, and store the received text, images, videos, voices, and/or actions in the memory 312 or use the communication module 316 and It can be provided to the information processing system 230 through the network 220. For example, the processor 314 receives a user's input requesting street view images and aerial images for a specific area and provides them to the information processing system 230 through the communication module 316 and the network 220. can do.

The processor 314 of the user terminal 210 manages, processes, and/or stores information and/or data received from the input/output device 320, other user terminals, the information processing system 230, and/or a plurality of external systems. It can be configured to do so. Information and/or data processed by processor 314 may be provided to information processing system 230 via communication module 316 and network 220. The processor 314 of the user terminal 210 may transmit information and/or data to the input/output device 320 through the input/output interface 318 and output the information. For example, the processor 314 may display the received information and/or data on the screen of the user terminal.

The processor 334 of the information processing system 230 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals 210 and/or a plurality of external systems. Information and/or data processed by the processor 334 may be provided to the user terminal 210 through the communication module 336 and the network 220.

FIG. 4 is a diagram illustrating an example of determining a matching target building among a plurality of buildings included in a street view image captured from the ground according to an embodiment of the present disclosure. The information processing system can determine a matching target building to acquire a building control point among a plurality of buildings included in a street view image taken from the ground. Here, the street view image may be a 360-degree panoramic image captured using a vehicle equipped with at least one camera. That is, the street view image may be a panoramic image created using equirectangular projection. Additionally, street view images may include low-precision GPS location information acquired when shooting.

According to one embodiment, the information processing system may determine a matching target building among a plurality of buildings included in the street view image based on a street view image for a specific area and a 3D model for the specific area. Here, the 3D model may include 3D building models for a plurality of buildings.

As a specific example, a matching target building may be determined through the first state 410 and the second state 420. The first state 410 represents an example of a result of projecting a 3D building model for a plurality of buildings onto a street view image. According to one embodiment, the information processing system may project a 3D building model for a plurality of buildings onto the street view image using the location information and direction information of the street view image. For example, as shown, a plurality of 3D absolute coordinate position information associated with a 3D building model for each of a plurality of buildings (e.g., 3D absolute coordinate position information of a vertex of the 3D building model) ) can be converted into a plurality of two-dimensional points on the spherical coordinate system of the street view image and projected onto the street view image.

The second state 420 represents an example of a result in which an area associated with each of a plurality of buildings in the street view image is determined based on the projection result. According to one embodiment, the information processing system may determine an area associated with each of a plurality of buildings in the street view image based on the projection result. For example, the information processing system may determine a minimum image area including a plurality of two-dimensional points for each of a plurality of buildings as an area associated with the building. Here, the minimum image area can be determined using the Convex Hull algorithm, etc.

Then, the information processing system may determine the matching target building by excluding buildings that are obscured by other buildings, based on the area associated with each of the plurality of buildings. To determine which building is obscured by another building, 3D absolute coordinate location information included in the 3D building model can be used. For example, determine the distance from the location where the street view image was taken to each building, and based on that distance, match the distant building if the area of the building located further away is obscured by the area of the building located closer. It can be excluded from the target building.

For example, as shown, based on the projection result, a minimum image area containing a two-dimensional point for each of a plurality of buildings may be determined as an area associated with the building. That is, the first building area 422, the second building area 424, the third building area 426, and the fourth building area 428 may be determined based on two-dimensional points associated with different buildings. The information processing system then controls the first building area 422, the second building area 424, and the third building area that are not obscured by other buildings, except for the fourth building area 428 that is obscured by the other building. Area 426 may be determined as the matching target building.

According to one embodiment, the information processing system may determine an aerial image suitable for performing feature matching with a street view image including a matching target building in order to obtain a building control point. Specifically, the information processing system may receive a plurality of aerial images including the face of a matching target building included in the street view image. Additionally, the information processing system may determine the aerial image having the highest similarity to the viewpoint from which the matching target building is viewed in the street view image among the plurality of aerial images as the aerial image associated with the street view image. For example, the information processing system can determine the normal vector on the surface of the three-dimensional building model of the building to be matched. Here, the normal vector on a surface in the 3D building model of the matching target building may represent the average of normal vectors for a plurality of surfaces in the 3D building model of the matching target building. The information processing system may determine, for each of the plurality of aerial images, a straight line vector connecting one point on the surface of the matching target building from the point of capture. Then, the information processing system may determine the aerial image with the highest dot product between the normal vector and the straight line vector as the aerial image associated with the street view image. The method of determining the aerial image having the highest similarity to the viewpoint from which the matching target building is viewed in the street view image is not limited to the above-described example, and may be determined by various methods.

Figure 4 shows that the first to fourth building areas 422 to 428 are determined among a plurality of buildings in the street view image, but this is for convenience of explanation, and the information processing system is included in the street view image. Each building area can be determined for all of the plurality of buildings, and among them, the building that is not obscured by other buildings can be determined as the matching target building.

FIG. 5 is a diagram illustrating an example of acquiring a building control point 560 by performing feature matching between an aerial image 520 and a street view image 540 according to an embodiment of the present disclosure. Here, the street view image 540 may be a 360-degree panoramic image captured using a vehicle equipped with at least one camera and generated using equirectangular projection. Additionally, the aerial image 520 may be an orthoimage. The aerial image 520 and the street view image 540 may include a plurality of buildings including a matching target building in a specific area. A specific method of selecting a matching target building in the aerial image 520 and the street view image 540 is described in detail with reference to FIG. 4.

Before performing feature matching for the building to be matched between the aerial image 520 and the street view image 540, in order to overcome the domain gap, the information processing system matches the aerial image 520 and its associated 3 Based on the dimensional model 510, a view synthesis image 530 having the same viewpoint as that of the street view image 540 may be generated. Here, the viewpoint composite image 530 may be a flat image. Additionally, the information processing system can convert the street view image 540 in a spherical coordinate system into a flat image 550 in a plane coordinate system using a perspective projection method. Specifically, the direction and size of the matching target building can be determined in the spherical coordinate system of the street view image 540. And, based on the determined direction and size of the matching target building, the information processing system can convert the street view image 540 into a planar image 550 including the area of the matching target building. Then, the information processing system may extract a plurality of building control points 560 by performing feature matching between the viewpoint composite image 530 and the street view plan image 550.

The building control point 560 extracted by performing feature matching between the viewpoint composite image 530 and the street view plan image 550 may mean a corresponding pair of points in the matching target building. Specifically, the building control point 560 is the point of the matching target building in the street view image 540 (e.g., (u1, v1)) and the point of the matching target building in the aerial image 520 (e.g., For example, it can represent a corresponding pair of (x1, y1, z1)), and each building control point can have 3D absolute coordinate position information.

In Figure 5, it is shown that a plurality of building control points 560 are extracted by performing feature matching between one street view image 540 and the corresponding aerial shot image 520, but the information processing system uses multiple street views. By performing feature matching on video-aerial photography pairs, a large number of building control points can be automatically extracted.

According to the conventional method, in order to estimate absolute coordinate location information for a specific point in the image, the operator manually performs ground point surveying or manually determines the location of the reference point in the image to use a standard reference point provided by the country. Tagging work was performed. This method has the problem that reference point surveying or tagging work is inefficient, and costs increase as the scope of work expands. According to the method of the present disclosure, a large number of building control points can be automatically extracted using matching information between the street view image 540 and the aerial image 520 without additional reference point surveying or tagging work, and the ground Even in areas where it is difficult to recognize the ground due to multiple vehicles being located, a large number of building control points can be extracted using matching information between buildings commonly included in the street view image (540) and the aerial image (520). This allows limited resources to be used efficiently.

FIG. 6 is a block diagram showing a specific method of acquiring a building control point 670 by performing feature matching 650 between a 3D model 620 and street view data 610 according to an embodiment of the present disclosure. . Here, the street view data 610 is a plurality of street view images captured from a plurality of nodes within a specific area, and may be a panoramic image generated using equirectangular projection. The 3D model 620 may be a model for a specific area created based on the aerial image 622. As shown, the 3D model 620 may include a plurality of aerial images 622 and 3D absolute coordinate location information for a specific area. For example, the 3D absolute coordinate location information 624 may include a 3D model for a specific area containing 3D geometric information expressed in absolute position, a 3D building model, a digital elevation model (DEM), etc. there is.

According to one embodiment, the information processing system may receive a street view image 612 captured from the ground from the street view data 610. Here, the street view image 612 may include a plurality of buildings including the matching target building. The expression method representing a specific point in the street view image 612 is based on the format of the street view image 612 (e.g., equirectangular format, cubic format, projective format, etc.). It may vary depending on.

According to one embodiment, the information processing system may convert at least a portion of the street view image 612 into a planar image 614 using a perspective projection method. Here, the flat image 614 may be an image related to the matching target building included in the street view image 612. Specifically, the information processing system can convert a street view image 612 of a spherical coordinate system for a matching target building into a flat image 614 of a planar coordinate system. The specific method of converting the flat image 614 using the perspective projection method is not limited to a specific method, and any image conversion method (Inverse Perspective Mapping (IPM), Homography Transform technique, etc.) ) can be used. The information processing system may then extract a first set of feature points 630 associated with the building to be matched from the planar image 614 .

According to one embodiment, the information processing system may receive an aerial image 622 associated with the street view image 612 from the 3D model 620. Here, the aerial image 622 is an aerial image having the highest similarity to the viewpoint from which the matching target building is viewed in the street view image 612 among a plurality of aerial images including the face of the matching target building. It may be an aerial footage determined by video.

According to one embodiment, the information processing system may generate a viewpoint composite image 626 for the matching target building based on the aerial image 622 and the 3D absolute coordinate location information 624. Specifically, the information processing system is based on an aerial image of a specific area and the 3D absolute coordinate location information 624 associated therewith, and provides a viewpoint that has the same viewpoint as that of the street view image 612 or the planar image 614. A composite image 626 can be generated. For example, the information processing system may generate a viewpoint composite image for an overlapping area of a matching target building that can be identified in both the street view image 612/planar image 614 and the aerial image 622.

Specifically, the information processing system uses the location information and direction information of the street view image 612 to project a 3D building model for the matching target building onto the street view image 612 to create a first depth-map. ) can be created. In addition, the information processing system uses the location information and direction information of the aerial image 622 to project a 3D building model for the matching target building onto the aerial image 622 to generate a second depth map. . Then, the information processing system may determine an overlapping area of the matching target building that can be identified in both the street view image 612 and the aerial image 622 based on the first depth map and the second depth map. Afterwards, the information processing system can use the 3D model to create a viewpoint composite image having the same viewpoint as the viewpoint of the street view image 612 for the overlapping area of the matching target building included in the aerial photography image 622. there is. Then, the information processing system may extract a second set of feature points 640 associated with the matching target building from the viewpoint composite image 626. The information processing system may then perform feature matching 650 between the first set of feature points 630 and the second set of feature points 640.

According to one embodiment, the information processing system performs feature matching 650 to extract a plurality of point correspondence pairs 660 between the planar image 614 and the viewpoint composite image 626 and store them as feature matching results. there is. Then, based on the extracted plurality of point correspondence pairs 660, the information processing system generates a first set of matching pixel position information 662 within the planar image 614 and a first set of matching pixel position information 662 within the viewpoint composite image 626. A second set of matching pixel location information 664 may be identified.

The information processing system may convert a first set of matching pixel location information 662 within the planar image 614 into a third set of matching pixel location information 666 within the street view image. In this case, the information processing system may use inversion of the perspective projection method (eg, homography transform) used to convert the street view image 612 into the flat image 614. Additionally, the information processing system may convert the second set of matching pixel location information 664 in the viewpoint composite image 626 into a fourth set of matching pixel location information in the aerial image. And, the information processing system may acquire a plurality of three-dimensional absolute coordinate position information 668 associated with the fourth set of matching pixel positions in the aerial image. The plurality of 3D absolute coordinate position information 668 may be 3D absolute coordinate position information for each of the fourth set of matching pixel positions. Then, the information processing system associates the third set of matching pixel location information 666 and the plurality of three-dimensional absolute coordinate position information 668 to form a plurality of two-dimensional point-three-dimensional point correspondence pairs for the matching target building, That is, the building control point 670 can be determined.

According to one embodiment, the information processing system may additionally extract a plurality of ground control points along with the building control point 670 based on the 3D model 620 and the street view data 610. Here, the ground control point represents a corresponding pair of a point on the ground of a street view image and a point on the ground of an aerial shot image, and each ground control point may have 3D absolute coordinate position information. For example, the information processing system may convert the street view image 612 into a top view image and extract a third set of feature points from the top view image. Additionally, the information processing system may extract a fourth set of feature points from the aerial image 622. Here, the aerial image 622 may be a true-ortho image. Then, the information processing system may perform feature matching between the third set of feature points and the fourth set of feature points to further obtain a plurality of ground control points.

FIG. 7 is a diagram illustrating an example of an image 710 in which a viewpoint composite image associated with an aerial shot image is projected onto a planar image associated with a street view image according to an embodiment of the present disclosure. The street view image may include absolute coordinate location information and direction information where the image was captured. For example, location information included in a street view image may be low-accuracy location information obtained using GPS equipment when shooting a street view image at a specific node. Additionally, a 3D model for a specific area may include high accuracy 3D geometric information expressed in absolute position.

The image shown in FIG. 7 shows an example of an image 710 in which a viewpoint composite image associated with an aerial shot image is projected onto a planar image associated with a street view image. The shape of the building including the texture of the building in the projected image 710 may represent a matching target building based on low-accuracy location information and direction information of the street view image. Additionally, the shaded silhouette of the building in the projected image 710 may represent the silhouette of the matching target building based on high-accuracy absolute coordinate location information included in the 3D building model. Referring to FIG. 7 , it can be seen that the shape of the building including the texture of the building in the projected image 710 and the silhouette of the building displayed in shading do not exactly match. This may be due to the difference in accuracy between the location information included in the street view image and the location information included in the 3D model.

The problem that arises due to the low-accuracy location information of the street view image is that the absolute coordinate location information of the street view image is obtained by using the map matching point (e.g. ground control point, building control point, etc.)/map matching line described above. This can be solved by estimating/correcting the and direction information.

FIG. 8 is a diagram illustrating an example of a matching result between a viewpoint composite image 810 and a planar image 820 according to an embodiment of the present disclosure. According to one embodiment, the information processing system extracts a plurality of point correspondence pairs for the matching target building between the viewpoint composite image 810 and the planar image 820 according to the method described in FIGS. 5 and 6 and provides a matching result. You can save it. Here, the viewpoint composite image 810 and the planar image 820 may be different images of the same matching target building. Specifically, the viewpoint composite image 810 may be an image of a matching target building created based on an aerial image, and the planar image 820 may be an image of a matching target building created based on a street view image.

As shown, the information processing system provides a plurality of point correspondence pairs between the first set of feature points extracted from the planar image 820 and the second set of feature points extracted from the viewpoint composite image 810 in association with the matching target building. can be extracted and saved as a feature matching result for the matching target building. Here, the first set of feature points and the second set of feature points may be extracted using the same or the same type of feature extractor. The feature extractor is a visual feature extraction neural network model of the corresponding pixels in the planar image 820 generated by converting the street view image and the viewpoint composite image 810 generated based on the aerial image. It may be a model learned so that visual feature descriptors are similarly extracted. Additionally, the method of performing feature matching is not limited to a specific method, and any feature matching method (SuperPoint/Glue, deep learning-based feature matching method such as R2D2, etc.) can be used.

FIG. 9 is a diagram illustrating an example of a plurality of building control points matched between a street view image 910 and an aerial image 920 for a specific matching target building according to an embodiment of the present disclosure. According to one embodiment, the information processing system generates matching pixel location information in the street view image 910 associated with the planar image and aerial photography image 920 associated with the viewpoint composite image, based on the extracted plurality of point correspondence pairs. It can be converted into matching pixel location information. In addition, the information processing system associates a plurality of 3D absolute coordinate location information associated with the matching pixel location information in the aerial shot image 920 with the matching pixel location information in the street view image 910, thereby providing information about the matching target building. A plurality of 2D point-3D point correspondence pairs, that is, a plurality of building control points, can be determined.

FIG. 9 illustrates an example of a plurality of 2D point-3D point correspondence pairs for one building between one street view image 910 and one aerial photography image 920, but the present invention is not limited thereto. For example, a plurality of 2D point-3D point correspondence pairs for a plurality of buildings between one street view image 910 and one aerial capture image 920 may be acquired.

Figure 10 is a flowchart illustrating an example of a method for automatically acquiring a building control point according to an embodiment of the present disclosure. The method 1000 may be initiated by a processor (eg, at least one processor of an information processing system) receiving a street view image captured from the ground (S1010). Here, the street view image may include a plurality of buildings including the matching target building, and the street view image may be a panoramic image generated by equirectangular projection. Additionally, the processor may receive a 3D model for a specific area including 3D geometric information expressed as an absolute position (S1020). Multiple buildings may be located within a specific area. The 3D model may include 3D building models for multiple buildings.

According to one embodiment, the processor may determine some of the plurality of buildings included in the street view image as the matching target building. Specifically, the processor may project a 3D building model for a plurality of buildings onto the street view image using the location information and direction information of the street view image. For example, using the location information and direction information of the street view image, a plurality of 3D absolute coordinate position information associated with a 3D building model for each of a plurality of buildings is converted into a plurality of 2D points on the spherical coordinate system of the street view image. It can be converted to . And, based on the projection result, the processor may determine an area associated with each of the plurality of buildings in the street view image. For example, the processor may determine a minimum image area including a plurality of two-dimensional points for each of a plurality of buildings as an area associated with the building. Thereafter, the processor may determine a matching target building by excluding buildings that are obscured by other buildings, based on the area associated with each of the plurality of buildings.

Then, the processor may receive an aerial image associated with the street view image (S1030). Here, the aerial image may include a matching target building. In one embodiment, the processor may receive a plurality of aerial images including the face of a matching target building included in the street view image. Additionally, the processor may determine, among the plurality of aerial images, the aerial image that has the highest similarity to the viewpoint from which the matching target building is viewed in the street view image as the aerial image associated with the street view image. For example, the processor can determine the normal vector on the surface of the 3D building model of the building to be matched. Additionally, for each of the plurality of aerial images, the processor may determine a straight line vector connecting one point on the surface of the matching target building from the point of capture. Afterwards, the processor may determine the aerial image with the highest inner product value between the normal vector and the straight line vector as the aerial image associated with the street view image. In another example, when the matching target building included in the street view image includes a plurality of surfaces, the processor may represent the average of the normal vectors on the multiple surfaces in the three-dimensional building model of the matching target building. In one embodiment, the processor may determine a plurality of aerial images having a viewpoint that is highly similar to the viewpoint from which the matching target building is viewed in the street view image.

According to one embodiment, the processor may generate a view synthesis image having the same viewpoint as that of the street view image, based on the 3D model and the aerial image. Here, the viewpoint composite image may include a matching target building. In one embodiment, the processor may determine an overlapping area of a matching target building that can be identified in both the street view image and the aerial image. For example, the processor may use the location information and direction information of the street view image to generate a first depth map (depth-map) by projecting a 3D building model for the matching target building onto the street view image. Additionally, the processor may generate a second depth map by using the location information and direction information of the aerial image to project a 3D building model for the matching target building onto the aerial image. Thereafter, the processor may determine an overlapping area of the matching target building that can be identified in both the street view image and the aerial image, based on the first depth map and the second depth map. Then, the processor may use the 3D model to generate a viewpoint composite image having the same viewpoint as the viewpoint of the street view image of the overlapping area of the matching target building included in the aerial image.

The processor may acquire a plurality of building control points associated with the matching target building based on the street view image and the aerial image (S1040). Here, each building control point may represent a corresponding pair of one point on the matching target building in the street view image and one point on the matching target building in the aerial image, and each building control point may contain three-dimensional absolute coordinate location information. You can have it. Additionally, a plurality of building control points associated with the matching target building may be acquired based on the street view image and the viewpoint composite image.

According to one embodiment, the processor may perform feature matching based on the street view image and the viewpoint composite image. For example, the processor may convert at least a portion of the street view image into a flat image using a perspective projection method. Here, the flat image may include a matching target building. Additionally, the processor may detect a first set of feature points associated with the matching target building from the planar image. Additionally, the processor may detect a second set of feature points associated with the matching target building from the viewpoint composite image. Here, the first set of feature points and the second set of feature points may be extracted using the same type of visual feature extractor. Then, the processor may extract a plurality of point correspondence pairs between the first set of feature points and the second set of feature points and store them as feature matching results. For example, the processor may identify a first set of matching pixel location information in a planar image and a second set of matching pixel location information in a viewpoint composite image, based on the extracted plurality of point correspondence pairs. . And, the processor may convert the first set of matching pixel location information in the planar image into a third set of matching pixel location information in the street view image. Additionally, the processor may convert a second set of matching pixel location information in the viewpoint composite image into a fourth set of matching pixel location information in the aerial image. Additionally, the processor may obtain a plurality of 3D absolute coordinate position information associated with the fourth set of matching pixel positions in the aerial image. Thereafter, the processor may associate the second set of matching pixel location information and the plurality of 3D absolute coordinate location information to determine a plurality of 2D point-3D point correspondence pairs for the matching target building. The processor may acquire a plurality of building control points based on the feature matching results.

According to one embodiment, the processor may acquire a plurality of ground control points along with a plurality of building control points. Specifically, the processor can acquire a plurality of ground control points (GCP) based on street view images and aerial images. Here, the aerial image may be a true-ortho image. Additionally, each ground control point may represent a corresponding pair of a point on the ground of a street view image and a point on the ground of an aerial image. Additionally, each ground control point may have 3D absolute coordinate position information. For example, the processor can convert a street view image into a top view image. And, the processor can extract a third set of feature points from the top view image. Additionally, the processor may extract a fourth set of feature points from the aerial image. Then, the processor may perform feature matching between the third set of feature points and the fourth set of feature points to obtain a plurality of ground control points.

The flowchart of FIG. 10 and the above description are only examples, and the scope of the present disclosure is not limited thereto. For example, at least one step may be added/changed/deleted, or the order of each step may be changed.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution on a computer. The medium may continuously store a computer-executable program, or may temporarily store it for execution or download. In addition, the medium may be a variety of recording or storage means in the form of a single or several pieces of hardware combined. It is not limited to a medium directly connected to a computer system and may be distributed over a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be something configured to store program instructions, including ROM, RAM, flash memory, etc. Additionally, examples of other media include recording or storage media managed by app stores that distribute applications, sites or servers that supply or distribute various other software, etc.

The methods, operations, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those skilled in the art will understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchange of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques may include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logical blocks, modules, and circuits described in connection with this disclosure may be general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or It may be implemented or performed as any combination of those designed to perform the functions described in. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

For firmware and/or software implementations, techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), and PROM ( on computer-readable media such as programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may also be implemented as stored instructions. Instructions may be executable by one or more processors and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the above-described embodiments have been described as utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the disclosure is not limited thereto and may also be implemented in conjunction with any computing environment, such as a network or distributed computing environment. . Furthermore, aspects of the subject matter of this disclosure may be implemented in multiple processing chips or devices, and storage may be similarly effected across the multiple devices. These devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in relation to some embodiments in this specification, various modifications and changes may be made without departing from the scope of the present disclosure as can be understood by a person skilled in the art to which the invention pertains. Additionally, such modifications and changes should be considered to fall within the scope of the claims appended hereto.

Claims

In a method of automatically acquiring a Building Control Point (BCP), performed by at least one processor,

Receiving a street view image taken from the ground, wherein the street view image includes a plurality of buildings including a matching target building;

Receiving an aerial image associated with the street view image, wherein the aerial image includes the matching target building; and

Based on the street view image and the aerial image, acquiring a plurality of building control points associated with the matching target building.

Method for automatically acquiring building control points, including:
According to paragraph 1,

Each building control point represents a corresponding pair of one point in the matching target building in the street view image and one point in the matching target building in the aerial image,

A method of automatically acquiring building control points, where each building control point has 3D absolute coordinate location information.
According to paragraph 1,

Receiving a 3D model for a specific area including 3D geometric information expressed in absolute positions, wherein the plurality of buildings are located within the specific area; and

Based on the 3D model and the aerial image, generating a view synthesis image having the same viewpoint as that of the street view image, wherein the viewpoint synthesis image includes the matching target building.

It further includes,

A method of automatically acquiring building control points, wherein a plurality of building control points associated with the matching target building are acquired based on the street view image and the viewpoint composite image.
According to claim 1,

The street view image is a panoramic image generated by equirectangular projection. A method of automatically acquiring building control points.
According to paragraph 3,

The three-dimensional model includes a three-dimensional building model for the plurality of buildings,

The method is:

Projecting a 3D building model for the plurality of buildings onto the street view image using the location information and direction information of the street view image;

Based on the projection result, determining an area associated with each of the plurality of buildings in the street view image; and

Determining the matching target building by excluding buildings that are obscured by other buildings, based on areas associated with each of the plurality of buildings.

A method for automatically acquiring building control points, further comprising:
According to clause 5,

The projection step is,

Using the location information and direction information of the street view image, a plurality of 3D absolute coordinate location information associated with a 3D building model for each of the plurality of buildings is converted to a plurality of 2D points on the spherical coordinate system of the street view image. steps to convert

Method for automatically acquiring building control points, including:
According to clause 6,

The step of determining an area associated with each of the plurality of buildings includes:

Determining a minimum image area including a plurality of two-dimensional points for each of the plurality of buildings as an area associated with the building.

Method for automatically acquiring building control points, including:
According to paragraph 1,

The step of receiving an aerial image related to the street view image is,

Receiving a plurality of aerial images including the face of a matching target building included in the street view image; and

Among the plurality of aerial images, determining the aerial image having the highest similarity to the viewpoint from which the matching target building is viewed in the street view image as the aerial image associated with the street view image.

Method for automatically acquiring building control points, including:
According to clause 8,

The step of determining an aerial image related to the street view image is,

determining a normal vector on the surface in a 3D building model of the matching target building;

For each of the plurality of aerial images, determining a straight line vector connecting one point on the surface of the matching target building from the point of capture; and

Determining the aerial image with the highest inner product value between the normal vector and the straight line vector as the aerial image associated with the street view image.

Method for automatically acquiring building control points, including:
According to paragraph 3,

The step of generating the viewpoint composite image is,

determining an overlapping area of a matching target building that can be identified in both the street view image and the aerial image; and

Using the 3D model, generating a viewpoint composite image having the same viewpoint as the viewpoint of the street view image of the overlapping area of the matching target building included in the aerial shot image.

Method for automatically acquiring building control points, including:
According to clause 10,

The step of determining the overlapping area of the matching target building is,

Using the location information and direction information of the street view image, projecting a 3D building model for the matching target building onto the street view image to generate a first depth map (depth-map);

Using the location information and direction information of the aerial image, projecting a 3D building model for the matching target building onto the aerial image to generate a second depth map; and

Based on the first depth map and the second depth map, determining an overlapping area of the matching target building that can be identified in both the street view image and the aerial image.

Method for automatically acquiring building control points, including:
According to paragraph 3,

The step of acquiring the plurality of building control points is:

performing feature matching based on the street view image and the viewpoint composite image; and

Acquiring a plurality of building control points based on the feature matching results

Method for automatically acquiring building control points, including:
According to clause 12,

The step of performing the feature matching is,

Converting at least a portion of the street view image into a planar image using a perspective projection method, wherein the planar image includes the matching target building;

detecting a first set of feature points associated with the matching target building from the planar image;

detecting a second set of feature points associated with the matching target building from the viewpoint composite image; and

Extracting a plurality of point correspondence pairs between the feature points of the first set and the feature points of the second set and storing them as feature matching results.

Method for automatically acquiring building control points, including:
According to clause 13,

The step of extracting the plurality of point correspondence pairs and storing them as feature matching results,

Identifying a first set of matching pixel location information in the planar image and a second set of matching pixel location information in the viewpoint composite image, based on the extracted plurality of point correspondence pairs.

Method for automatically acquiring building control points, including:
According to clause 14,

The step of extracting the plurality of point correspondence pairs and storing them as feature matching results,

converting a first set of matching pixel location information in the planar image into a third set of matching pixel location information in the street view image;

converting the second set of matching pixel location information in the viewpoint composite image into a fourth set of matching pixel location information in the aerial shot image;

acquiring a plurality of three-dimensional absolute coordinate position information associated with the fourth set of matching pixel positions in the aerial image; and

Determining a plurality of 2D point-3D point correspondence pairs for the matching target building by associating the second set of matching pixel location information and the plurality of 3D absolute coordinate location information.

A method for automatically acquiring building control points, further comprising:
According to clause 13,

A method for automatically acquiring building control points, wherein the first set of feature points and the second set of feature points are extracted using the same type of visual feature extractor.
According to paragraph 1,

Acquiring a plurality of ground control points (GCP) based on the street view image and the aerial image.

It further includes,

Each ground control point represents a corresponding pair of a point on the ground of the street view image and a point on the ground of the aerial image,

A method of automatically acquiring building control points, where each ground control point has 3D absolute coordinate location information.
According to clause 17,

The step of acquiring the plurality of ground control points is,

Converting the street view image into a top view image;

extracting a third set of feature points from the top-view image;

extracting a fourth set of feature points from the aerial image; and

Acquiring the plurality of ground control points by performing feature matching between the third set of feature points and the fourth set of feature points.

Including,

The aerial image is a true-ortho image. A method of automatically acquiring a building control point.
A computer-readable non-transitory recording medium recording instructions for executing the method according to claim 1 on a computer.
As an information processing system,

communication module;

Memory; and

At least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory

Including,

The at least one program is,

Receive a street view image taken from the ground - the street view image includes a plurality of buildings including a matching target building -

Receive an aerial image associated with the street view image, where the aerial image includes the matching target building,

An information processing system comprising instructions for acquiring a plurality of building control points associated with the matching target building based on the street view image and the aerial image.