KR102556765B1 - Apparatus and method for visual localization - Google Patents

Abstract

A method performed by a computing device is disclosed. The method includes: receiving a query image and additional information of the device from a device of a client; extracting first keypoints for the query image from the query image using an artificial intelligence-based keypoints extraction model; generating a first detection result corresponding to a predetermined movable object in the query image from the query image by using an artificial intelligence-based movable object detection model; Determining 1-2 feature points removed from the query image and 1-1 feature points maintained on the query image among the first feature points based on a comparison result between the first detection result and the first feature points doing; and performing visual localization on the device capturing the query image, based at least in part on 1-1 feature points on the query image and additional information of the device. .

Description

Method and apparatus for visual localization {APPARATUS AND METHOD FOR VISUAL LOCALIZATION}

The present disclosure relates to image processing and more specifically to visual localization.

As demand for location-based services increases, the need for accurate location information has increased. The most common way to determine location on mobile devices and mobile platforms is the Global Navigation Satellite System (GNSS). However, since the GNSS signal can be blocked by obstacles in an indoor environment, there is a limitation that it can be easily used only in an outdoor environment.

Although various technologies for performing indoor location recognition have been proposed, it is a reality that many indoor location recognition techniques do not exceed a fingerprint-based location recognition algorithm using a wireless signal. In this method, collected data related to Wi-Fi RSS (received signal strength) or MFS (magnetic field strength) is compared with data from a fingerprinting database. The fingerprinting-based system has the advantage of being easy to build, but it may be difficult to maintain good performance because the signal pattern itself is affected by changes in the system environment. Many alternatives, including Optical, RFID (Radio Frequency Identification), Bluetooth Beacons, ZigBee, and Pseudo Satellite, have been proposed to overcome the deficiencies of these fingerprinting-based systems, but even these alternatives cannot achieve high accuracy in complex indoor environments. It is rated as difficult.

Recently, as an alternative for realizing position estimation with high accuracy even in an indoor environment, research on a visual positioning system (VPS) has been actively conducted. The VPS technology may also be expressed as a visual localization technology, and this visual localization refers to a technology of estimating a current location or pose of a device using an image taken indoors or outdoors. Pose estimation of a device (or camera) is to determine translation and rotation information of a dynamically changing camera viewpoint. Such camera pose estimation technology is being used in various applications such as mixed reality, augmented reality, robot navigating, and 3-dimensional scene reconstruction.

Korean Registered Patent No. 10-2225093

The present disclosure has been made in response to the above background art, and is to efficiently use computing resources in camera pose estimation.

The present disclosure is to increase the accuracy of camera pose estimation and reduce the time required for camera pose estimation.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to some embodiments of the present disclosure for solving the above problems, a method performed by a computing device is disclosed. The method may include receiving a query image and additional information of the device from a device of a client; acquiring first keypoints for the query image from the query image by using an artificial intelligence-based keypoints acquisition model; obtaining a first detection result corresponding to a predetermined dynamic object in the query image from the query image by using an artificial intelligence-based dynamic object detection model; Determining 1-2 feature points removed from the query image and 1-1 feature points maintained on the query image among the first feature points based on a comparison result between the first detection result and the first feature points doing; and performing visual localization on the device capturing the query image, based at least in part on 1-1 feature points on the query image and additional information of the device. .

In one embodiment, the additional information of the device may include location information of the device. In one embodiment, when a record of estimating the past camera pose of the device exists, the additional information of the device is a preliminary for the camera pose of the device, calculated from a value of an Inertial Measurement Unit (IMU) of the device. It may contain estimation information.

In one embodiment, the step of performing visual localization on the device that has captured the query image includes: using additional information of the device, a candidate reference image that is a target of feature point matching among a plurality of reference images on the database determining them; and performing visual localization including pose estimation for a camera of the device based on feature point matching between the determined candidate reference images and the query image.

In one embodiment, the additional information of the device further includes camera information of the device, and based on the camera information of the device, a pose estimation algorithm used to perform the visual localization among a plurality of pose estimation algorithms. this can be determined.

In one embodiment, the method includes: recognizing first text existing in the query image from the query image using an AI-based text recognition model; and a 1-2 text removed from the query image and a 1-1 text maintained on the query image among the first texts based on a comparison result of the first detection result and the recognized first text. A determining step may be further included.

In one embodiment, the step of determining the 1-2 text removed from the query image and the 1-1 text maintained on the query image of the first text may include the predetermined text included in the first detection result. The method may include determining first text included in a detection region corresponding to a dynamic object as first-second text to be removed from the query image.

In one embodiment, the method may further include detecting, from the query image, a first predetermined landmark present in the query image by using an artificial intelligence-based landmark detection model. . In one embodiment, the step of performing visual localization on the device capturing the query image may include at least partially applying 1-1 feature points on the query image, additional information of the device, and the first landmark. Based on the method, visual localization may be performed on the device that has captured the query image.

In one embodiment, the step of performing visual localization on the device that has captured the query image: using the additional information of the device and the first landmark, among a plurality of reference images on the database, a target of feature point matching determining candidate reference images to be; and performing visual localization including pose estimation for a camera of the device based on feature point matching between the determined candidate reference images and the query image.

In one embodiment, performing visual localization on the device that has captured the query image may include a reference image stored in a database, 1-1 feature points on the query image, and additional information of the device, and performing visual localization on the device that has captured the query image.

In one embodiment, the performing of visual localization on the device that has captured the query image includes matching between first feature points of the query image and second feature points of the reference image among a plurality of reference images. determining relative camera pose information for the query image according to a comparison result between 3D camera coordinates assigned to the pose reference image determined based on the comparison and 2D coordinates of 1-1 feature points of the query image; determining absolute camera pose information of the query image with respect to an origin based on 3D camera coordinates assigned to the pose reference image and the relative camera pose information; and performing visual localization on the device that has captured the query image based on the absolute camera pose information.

In one embodiment, extracting first feature points of the query image from the query image includes obtaining pixel coordinates and descriptors corresponding to the first feature points, And the step of performing visual localization on the device capturing the query image may include comparing descriptors corresponding to the first feature points of the query image with descriptors corresponding to second feature points of a reference image stored in a database. , performing visual localization on the device that captured the query image.

In one embodiment, performing visual localization on the device that has captured the query image includes descriptors corresponding to the first feature points of the query image and second feature points of a reference image stored in a database. determining a pose reference image for visual localization for the device from among a plurality of reference images by comparing descriptors; And based on 3D coordinate information mapped to the pose reference image, matching information between the first feature points and the second feature points, and pixel coordinates corresponding to the first feature points of the query image, information about the device is determined. It may include performing visual localization including camera pose estimation.

In one embodiment, the method further comprises: extracting second feature points for the reference image from the reference image using the feature point acquisition model; obtaining a second detection result corresponding to a predetermined dynamic object in the reference image from the reference image by using the dynamic object detection model; Determining 2-2 feature points removed from the reference image and 2-1 feature points maintained on the reference image among the second feature points based on a comparison result between the second detection result and the second feature points doing; and acquiring 3D camera coordinates corresponding to the 2-1 feature points from a predefined 3D map.

In one embodiment, based on a comparison result between the first detection result and the first feature points, 1-2 feature points removed from the query image among the first feature points and a first feature point maintained on the query image The step of determining -1 feature points may include determining first feature points included in a detection area corresponding to the predetermined dynamic object included in the first detection result as 1-2 feature points removed from the query image, and and determining first feature points not included in a detection area corresponding to the predetermined dynamic object included in the first detection result as 1-1st feature points maintained on the query image.

In one embodiment, a computer program stored on a computer readable storage medium is disclosed. The computer program, when executed by a computing device, causes the computing device to perform the following operations: receiving a query image and additional information of the device from a device of a client; extracting first feature points of the query image from the query image by using an artificial intelligence-based feature point acquisition model; obtaining a first detection result corresponding to a predetermined dynamic object in the query image from the query image by using an artificial intelligence-based dynamic object detection model; Determining 1-2 feature points removed from the query image and 1-1 feature points maintained on the query image among the first feature points based on a comparison result between the first detection result and the first feature points action; and performing visual localization on the device that has captured the query image, based at least in part on 1-1 feature points on the query image and additional information of the device.

In one embodiment, a computing device is disclosed. The computing device may include at least one processor; and memory. The at least one processor may perform: receiving a query image and additional information of the device from a device of a client; extracting first feature points of the query image from the query image by using an artificial intelligence-based feature point acquisition model; obtaining a first detection result corresponding to a predetermined dynamic object in the query image from the query image by using an artificial intelligence-based dynamic object detection model; Determining 1-2 feature points removed from the query image and 1-1 feature points maintained on the query image among the first feature points based on a comparison result between the first detection result and the first feature points action; and performing visual localization on the device that captured the query image, based at least in part on the 1-1 feature points on the query image and the additional information of the device.

Computing resources may be efficiently used in camera pose estimation according to an embodiment of the present disclosure.

An embodiment of the present disclosure can increase the accuracy of camera pose estimation and reduce the time required for camera pose estimation.

Effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. .

Various aspects are now described with reference to the drawings, wherein like reference numbers are used to collectively refer to like elements. In the following embodiments, for explanation purposes, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspect(s) may be practiced without these specific details.
1 schematically illustrates a block configuration diagram of a computing device according to an embodiment of the present disclosure.
2 is a schematic diagram exemplarily illustrating a network function according to an embodiment of the present disclosure.
3 exemplarily illustrates a method of performing visual localization for a device that captures a query image according to an embodiment of the present disclosure.
4 exemplarily illustrates a method of storing metadata mapped to a reference image in order to perform visual localization according to an embodiment of the present disclosure.
5 exemplarily illustrates an operation of a feature point acquisition model according to an embodiment of the present disclosure.
6 exemplarily illustrates operations of a dynamic object detection model and a landmark detection model according to an embodiment of the present disclosure.
7 exemplarily illustrates an operation of a text recognition model according to an embodiment of the present disclosure.
8 illustratively illustrates a flow of image processing for efficiently performing visual localization according to an embodiment of the present disclosure.
9 illustratively illustrates a flow of image processing for efficiently performing visual localization according to an embodiment of the present disclosure.
10 exemplarily illustrates operations of models for performing visual localization according to an embodiment of the present disclosure.
11 depicts a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the present disclosure. However, it is apparent that these embodiments may be practiced without these specific details.

The terms “component,” “module,” “system,” and the like, as used herein, refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure, processor, object, thread of execution, program, and/or computer running on a processor. For example, both an application running on a computing device and a computing device may be components. One or more components may reside within a processor and/or thread of execution. A component can be localized within a single computer. A component may be distributed between two or more computers. Also, these components can execute from various computer readable media having various data structures stored thereon. Components may be connected, for example, via signals with one or more packets of data (e.g., data and/or signals from one component interacting with another component in a local system, distributed system) to other systems and over a network such as the Internet. data being transmitted) may communicate via local and/or remote processes.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless otherwise specified or clear from the context, “X employs A or B” is intended to mean one of the natural inclusive substitutions. That is, X uses A; X uses B; Or, if X uses both A and B, "X uses either A or B" may apply to either of these cases. Also, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the listed related items.

Also, the terms "comprises" and/or "comprising" should be understood to mean that the features and/or components are present. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, elements, and/or groups thereof. Also, unless otherwise specified or where the context clearly indicates that a singular form is indicated, the singular in this specification and claims should generally be construed to mean "one or more".

In addition, the term “at least one of A or B” should be interpreted as meaning “when only A is included”, “when only B is included”, and “when A and B are combined”.

Those skilled in the art will further understand that the various illustrative logical blocks, components, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or combinations of both. It should be recognized that it can be implemented as To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or as software depends on the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

The description of the presented embodiments is provided to enable any person skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art of this disclosure. The general principles defined herein may be applied to other embodiments without departing from the scope of this disclosure. Thus, the present invention is not limited to the embodiments presented herein. The present invention is to be accorded the widest scope consistent with the principles and novel features set forth herein.

In the present disclosure, terms expressed as Nth, such as first, second, or third, are used to distinguish a plurality of entities. For example, entities represented as first and second may be the same as or different from each other. The terms 1-1 and 1-2 and terms 2-1 and 2-2 may also be used to distinguish a plurality of entities from each other.

In the present disclosure, visual localization refers to a technique of estimating a current location or pose of a device using an image captured indoors or outdoors. Pose estimation of a device (or camera) is to determine translation and rotation information of a dynamically changing camera viewpoint.

A query image in the present disclosure includes an image captured by a device, and a pose of a device corresponding to the query image is determined by using a reference image stored in a computing device (eg, a server, etc.) and the query image. can be determined or estimated.

A reference image in the present disclosure may include a pre-stored image used for estimating a pose of a device that has captured a query image. Reference images may include, for example, real images captured in a specific area to implement augmented reality or virtual reality. These reference images may be stored in the computing device 100 . Metadata mapped to the reference image may also be used for device pose estimation.

Metadata in the present disclosure may refer to additional information used in performing visual localization. For example, the metadata may include additional information other than the image, such as device-related information, pixel coordinate information of feature points, descriptors of feature points, landmark information obtained from an image, text information obtained from an image, and the like. . According to an embodiment of the present disclosure, comparison between a query image and a reference image can be made more smoothly by using various types of metadata, and furthermore, camera pose estimation can be made in an accurate manner using less computing resources. there is. For example, a candidate group of reference images to be matched with the query image may be reduced by utilizing metadata of the query image and/or the reference image. As another example, a more suitable camera pose estimation method may be used by utilizing metadata of a query image and/or a reference image.

1 schematically illustrates a block configuration diagram of a computing device 100 according to one embodiment of the present disclosure.

Computing device 100 according to an embodiment of the present disclosure may include a processor 110 and a memory 130 .

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include other components for performing a computing environment of the computing device 100, and only some of the disclosed components may constitute the computing device 100.

The computing device 100 in the present disclosure may mean any type of node constituting a system for implementing embodiments of the present disclosure. The computing device 100 may refer to any type of user terminal or any type of server. Components of the aforementioned computing device 100 are exemplary and some may be excluded or additional components may be included. For example, when the aforementioned computing device 100 includes a user terminal, an output unit (not shown) and an input unit (not shown) may be included within the scope of the computing device 100 .

A device of a client in the present disclosure may mean a device for generating a query image. A query image may be included in information captured by the client device. A query image captured by such a client device may be used for comparison with a reference image in a computing device (eg, a server), and thus a pose of the client device at a point of view may be determined.

Hereinafter, for convenience of explanation, a methodology for visual localization for estimating a pose of the client device by dividing the client device and the computing device 100 and receiving photographing information from the client device by the computing device 100 is described. will be explained However, an embodiment of performing visual localization in a client device generating a query image may also be included within the scope of the present disclosure. In this embodiment, a client device may serve as the computing device 100 .

The computing device 100 in the present disclosure may perform technical features according to embodiments of the present disclosure to be described later. For example, the computing device 100 may extract feature points for a query image from a query image using a feature point acquisition model, and may extract feature points for a query image from the query image using a dynamic object detection model. A detection result may be obtained, a feature point to be removed and a feature point to be retained may be determined on a query image by comparing the detection result and feature points, and visual localization may be performed on a device that captures the query image. For example, the computing device 100 obtains camera pose information including a reference image and the position and posture of a camera that has taken the reference image, and uses a feature point acquisition model to obtain feature points for the reference image from the reference image. extracting, using a dynamic object detection model, obtaining a detection result corresponding to a predetermined dynamic object in the reference image from a reference image, and referring among the feature points based on a comparison result between the detection result and the feature points. Determining feature points to be removed from an image and feature points maintained on a reference image, and metadata mapping coordinate information corresponding to the maintained feature points, descriptor information and camera pose information corresponding to the maintained feature points to the reference image can be saved as

In one embodiment, the processor 110 may include at least one core, and may include a central processing unit (CPU) or a general purpose graphics processing unit (GPGPU) of the computing device 100 . , a processor for data analysis and/or processing, such as a tensor processing unit (TPU).

Processor 110 may read a computer program stored in memory 130 to perform visual localization methodologies according to an embodiment of the present disclosure. In addition, the processor 110 may read a computer program stored in the memory 130 to perform visual localization methodologies according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning a neural network. The processor 110 is used for neural network learning, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating neural network weights using backpropagation. calculations can be performed. At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the network function. For example, the CPU and GPGPU can process learning of network functions and data classification using network functions. In addition, in one embodiment of the present disclosure, learning of a network function and data classification using a network function may be processed by using processors of a plurality of computing devices together. In addition, a computer program executed in a computing device according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program. Operations for a neural network according to an embodiment of the present disclosure will be described below with reference to FIG. 2 .

Additionally, the processor 110 may typically handle overall operations of the computing device 100 . For example, the processor 110 processes data, information, signals, etc. input or output through components included in the computing device 100 or drives an application program stored in a storage unit, thereby providing appropriate information or information to the user. A function can be provided or processed.

According to one embodiment of the present disclosure, memory 130 may store any type of information generated or determined by processor 110 and any type of information received by computing device 100 . According to one embodiment of the present disclosure, the memory 130 may be a storage medium that stores computer software that causes the processor 110 to perform operations according to embodiments of the present disclosure. Accordingly, the memory 130 may refer to computer readable media for storing software codes necessary for performing embodiments of the present disclosure, data subject to execution of the codes, and results of execution of the codes.

According to an embodiment of the present disclosure, the memory 130 may refer to any type of storage medium. For example, the memory 130 may be a flash memory type, a hard disk type ), multimedia card micro type, card type memory (eg SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only memory, ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. The computing device 100 may operate in relation to a web storage that performs a storage function of the memory 130 on the Internet. The description of the above memory is only an example, and the memory 130 used in the present disclosure is not limited to the above example.

The communication unit (not shown) in the present disclosure may be configured regardless of its communication mode, such as wired and wireless, and may be configured in various communication networks such as a personal area network (PAN) and a wide area network (WAN). can be configured. In addition, the network unit 150 may operate based on the known World Wide Web (WWW), and a wireless transmission technology used for short-range communication such as Infrared Data Association (IrDA) or Bluetooth. can also be used.

The computing device 100 in the present disclosure may include any type of user terminal and/or any type of server. Accordingly, embodiments of the present disclosure may be performed by a server and/or a user terminal.

A user terminal may include any type of terminal capable of interacting with a server or other computing device. User terminals include, for example, mobile phones, smart phones, laptop computers, personal digital assistants (PDAs), slate PCs, tablet PCs, and ultrabooks. can include

A device of a client in the present disclosure may include the aforementioned user terminal. The client's device may include modules for generating a depth map. Accordingly, the device may generate and transmit images and/or depth maps to computing device 100 . In one embodiment, a device of a client may mean any type of equipment for detecting an optical image and converting it into an electrical signal to transmit to the computing device 100 . For example, the client's device may include at least one of a camera, scanner, lidar, and/or vision sensor. The computing device 100 may include a device or may be interworked with an external device wirelessly or wired.

A server may include any type of computing system or computing device, such as, for example, microprocessors, mainframe computers, digital processors, portable devices and device controllers, and the like.

In an additional embodiment, the aforementioned server may include a storage unit (not shown) for storing and managing reference images, metadata corresponding to the reference images, and 3D maps. This storage may be included in the server or may exist under the management of the server. As another example, the storage unit may exist outside the server and may be implemented in a form capable of communicating with the server. In this case, the storage unit may be managed and controlled by another external server different from the server.

The computing device 100 according to some embodiments of the present disclosure may perform feature point acquisition, landmark detection, dynamic object detection, and/or text recognition using various artificial intelligence-based models. For example, the computing device 100 may acquire feature points corresponding to the query image by using a feature point acquisition model that is an artificial intelligence-based model.

In one embodiment, the feature point acquisition model may include a neural network built through deep learning or machine learning. The feature point acquisition model may acquire feature points from a query image and/or a depth map included in input data.

For example, the query image may include at least one red-green-blue (RGB) image and/or at least one grayscale image.

For example, the depth map may include an image or information indicating a relative distance of each pixel existing in a specific image. Accordingly, the depth map may include information related to a distance from a location where the query image is captured to the surface of the subject.

2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.

In the present disclosure, at least one of a feature point acquisition model, a dynamic object detection model, a landmark detection model, and/or a text recognition model may correspond to an artificial intelligence-based model.

Throughout this disclosure, artificial intelligence-based model, model, computational model, neural network, network function, and neural network may be used interchangeably. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes. Nodes (or neurons) constituting neural networks may be interconnected by one or more links.

In a neural network, one or more nodes connected through a link may form a relative relationship of an input node and an output node. The concept of an input node and an output node is relative, and any node in an output node relationship with one node may have an input node relationship with another node, and vice versa. As described above, an input node to output node relationship may be created around a link. More than one output node can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of data of the output node may be determined based on data input to the input node. Here, a link interconnecting an input node and an output node may have a weight. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. An output node value may be determined based on the weight.

As described above, in the neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship in the neural network. Characteristics of the neural network may be determined according to the number of nodes and links in the neural network, an association between the nodes and links, and a weight value assigned to each link. For example, when there are two neural networks having the same number of nodes and links and different weight values of the links, the two neural networks may be recognized as different from each other.

A neural network may be composed of a set of one or more nodes. A subset of nodes constituting a neural network may constitute a layer. Some of the nodes constituting the neural network may form one layer based on distances from the first input node. For example, a set of nodes having a distance of n from the first input node may constitute n layers. The distance from the first input node may be defined by the minimum number of links that must be passed through to reach the corresponding node from the first input node. However, the definition of such a layer is arbitrary for explanation, and the order of a layer in a neural network may be defined in a method different from the above. For example, a layer of nodes may be defined by a distance from a final output node.

An initial input node may refer to one or more nodes to which data is directly input without going through a link in relation to other nodes among nodes in the neural network. Alternatively, in a relationship between nodes based on a link in a neural network, it may mean nodes that do not have other input nodes connected by a link. Similarly, the final output node may refer to one or more nodes that do not have an output node in relation to other nodes among nodes in the neural network. Also, the hidden node may refer to nodes constituting the neural network other than the first input node and the last output node.

In the neural network according to an embodiment of the present disclosure, the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as the number of nodes progresses from the input layer to the hidden layer. can In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes of the input layer may be less than the number of nodes of the output layer and the number of nodes decreases as the number of nodes increases from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as the number of nodes increases from the input layer to the hidden layer. can A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the aforementioned neural networks.

A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can reveal latent structures in data. In other words, it can identify the latent structure of a photo, text, video, sound, or music (e.g., what objects are in the photo, what the content and emotion of the text are, what the content and emotion of the audio are, etc.). . Deep neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), an auto encoder, a restricted boltzmann machine (RBM), and a deep trust network ( It may include a deep belief network (DBN), a Q network, a U network, a Siamese network, a Generative Adversarial Network (GAN), and the like. The description of the deep neural network described above is only an example, and the present disclosure is not limited thereto.

The neural network may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Learning of the neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

A neural network can be trained in a way that minimizes output errors. In the learning of the neural network, the learning data is repeatedly input into the neural network, the output of the neural network for the training data and the error of the target are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. It is a process of updating the weight of each node of the neural network by backpropagating in the same direction. In the case of teacher learning, the learning data in which the correct answer is labeled is used for each learning data (ie, the labeled learning data), and in the case of comparative teacher learning, the correct answer may not be labeled in each learning data. That is, for example, learning data in the case of teacher learning regarding data classification may be data in which each learning data is labeled with a category. Labeled training data is input to a neural network, and an error may be calculated by comparing an output (category) of the neural network and a label of the training data. As another example, in the case of comparative history learning for data classification, an error may be calculated by comparing input learning data with a neural network output. The calculated error is back-propagated in a reverse direction (ie, from the output layer to the input layer) in the neural network, and the connection weight of each node of each layer of the neural network may be updated according to the back-propagation. The amount of change in the connection weight of each updated node may be determined according to a learning rate. The neural network's computation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently according to the number of iterations of the learning cycle of the neural network. For example, a high learning rate may be used in the early stage of neural network training to increase efficiency by allowing the neural network to quickly obtain a certain level of performance, and a low learning rate may be used in the late stage to increase accuracy.

In neural network learning, generally, training data can be a subset of real data (ie, data to be processed using the trained neural network), and therefore, errors for training data are reduced, but errors for real data are reduced. There may be incremental learning cycles. Overfitting is a phenomenon in which errors for actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that has learned a cat by showing a yellow cat does not recognize that it is a cat when it sees a cat other than yellow may be a type of overfitting. Overfitting can act as a cause of increasing the error of machine learning algorithms. Various optimization methods can be used to prevent such overfitting. To prevent overfitting, methods such as increasing the training data, regularization, inactivating some nodes in the network during learning, and using a batch normalization layer should be applied. can

A computer readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

Data structure can refer to the organization, management, and storage of data that enables efficient access and modification of data. Data structure may refer to the organization of data to solve a specific problem (eg, data retrieval, data storage, data modification in the shortest time). A data structure may be defined as a physical or logical relationship between data elements designed to support a specific data processing function. A logical relationship between data elements may include a connection relationship between user-defined data elements. A physical relationship between data elements may include an actual relationship between data elements physically stored in a computer-readable storage medium (eg, a persistent storage device). The data structure may specifically include a set of data, a relationship between data, and a function or command applicable to the data. Through an effectively designed data structure, a computing device can perform calculations while using minimal resources of the computing device. Specifically, the computing device can increase the efficiency of operation, reading, insertion, deletion, comparison, exchange, and search through an effectively designed data structure.

The data structure can be divided into a linear data structure and a non-linear data structure according to the shape of the data structure. A linear data structure may be a structure in which only one data is connected after one data. Linear data structures may include lists, stacks, queues, and decks. A list may refer to a series of data sets in which order exists internally. The list may include a linked list. A linked list may be a data structure in which data are connected in such a way that each data is connected in a single line with a pointer. In a linked list, a pointer can contain information about connection to the next or previous data. A linked list can be expressed as a singly linked list, a doubly linked list, or a circular linked list depending on the form. A stack can be a data enumeration structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (eg, inserted or deleted) at only one end of the data structure. The data stored in the stack may be a LIFO-Last in First Out (Last in First Out) data structure. A queue is a data listing structure that allows limited access to data, and unlike a stack, it can be a data structure (FIFO-First in First Out) in which data stored later comes out later. A deck can be a data structure that can handle data from either end of the data structure.

The nonlinear data structure may be a structure in which a plurality of data are connected after one data. The non-linear data structure may include a graph data structure. A graph data structure can be defined as a vertex and an edge, and an edge can include a line connecting two different vertices. A graph data structure may include a tree data structure. The tree data structure may be a data structure in which one path connects two different vertices among a plurality of vertices included in the tree. That is, it may be a data structure that does not form a loop in a graph data structure.

Throughout this specification, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, a neural network is unified and described. The data structure may include a neural network. And the data structure including the neural network may be stored in a computer readable medium. The data structure including the neural network may also include preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation function associated with each node or layer of the neural network, and neural network It may include a loss function for learning of . A data structure including a neural network may include any of the components described above. That is, the data structure including the neural network includes preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data obtained from the neural network, activation function associated with each node or layer of the neural network, and neural network. It may be configured to include all or any combination thereof, such as a loss function for learning of . In addition to the foregoing configurations, the data structure comprising the neural network may include any other information that determines the characteristics of the neural network. In addition, the data structure may include all types of data used or generated in the computational process of the neural network, but is not limited to the above. A computer readable medium may include a computer readable recording medium and/or a computer readable transmission medium. A neural network may consist of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network includes one or more nodes.

The data structure may include data input to the neural network. A data structure including data input to the neural network may be stored in a computer readable medium. Data input to the neural network may include training data input during a neural network learning process and/or input data input to a neural network that has been trained. Data input to the neural network may include pre-processed data and/or data subject to pre-processing. Pre-processing may include a data processing process for inputting data to a neural network. Accordingly, the data structure may include data subject to pre-processing and data generated by pre-processing. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

The data structure may include the weights of the neural network. (In this specification, weights and parameters may be used in the same meaning.) Also, a data structure including weights of a neural network may be stored in a computer readable medium. A neural network may include a plurality of weights. The weight may be variable, and may be changed by a user or an algorithm in order to perform a function desired by the neural network. For example, when one or more input nodes are interconnected by respective links to one output node, the output node is set to a link corresponding to values input to input nodes connected to the output node and respective input nodes. A data value output from an output node may be determined based on the weight. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

As a non-limiting example, the weights may include weights that are varied during neural network training and/or weights for which neural network training has been completed. The variable weight in the neural network learning process may include a weight at the time the learning cycle starts and/or a variable weight during the learning cycle. The weights for which neural network learning has been completed may include weights for which learning cycles have been completed. Accordingly, the data structure including the weights of the neural network may include a data structure including weights that are variable during the neural network learning process and/or weights for which neural network learning is completed. Therefore, it is assumed that the above-described weights and/or combinations of weights are included in the data structure including the weights of the neural network. The foregoing data structure is only an example, and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be stored in a computer readable storage medium (eg, a memory or a hard disk) after going through a serialization process. Serialization can be the process of converting a data structure into a form that can be stored on the same or another computing device and later reconstructed and used. A computing device may serialize data structures to transmit and receive data over a network. The data structure including the weights of the serialized neural network may be reconstructed on the same computing device or another computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network is a data structure for increasing the efficiency of operation while minimizing the resource of the computing device (for example, B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree). The foregoing is only an example, and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of the neural network. Also, the data structure including the hyperparameters of the neural network may be stored in a computer readable medium. A hyperparameter may be a variable variable by a user. Hyperparameters include, for example, learning rate, cost function, number of learning cycle iterations, weight initialization (eg, setting the range of weight values to be targeted for weight initialization), hidden unit number (eg, the number of hidden layers and the number of nodes in the hidden layer). The foregoing data structure is only an example, and the present disclosure is not limited thereto.

3 exemplarily illustrates a method of performing visual localization for a device that captures a query image according to an embodiment of the present disclosure.

The flowchart shown in FIG. 3 may be performed by, for example, computing device 100 .

In one embodiment, the computing device 100 may receive a query image and additional information of the device from a device of a client (310).

Computing device 100 may provide an augmented reality and/or virtual reality platform. Computing device 100 may allow the client to enjoy activities on the augmented reality and/or virtual reality platform by estimating the current location and orientation of the client device by obtaining data captured from the client's device.

In one embodiment, the query image may include a two-dimensional RGB image or grayscale image. In one embodiment, the query image may include a two-dimensional RGB image or grayscale image and a depth map. The additional information of the device may include location information of the device. Location information is, for example, geodetic datum or geodetic system (or geodetic reference datum, geodetic reference system, or geodetic reference frame), spatial reference system (SRS) or coordinate reference system (CRS) that are common in a particular region or country. , or coordinate values and/or direction values for the geographic coordinate system (GCS). Another example of location information may include GPS.

In addition, the additional device information may include additional device hardware-related information such as focal length, principal point, resolution, and the like. In addition, if there is a record of estimating the camera pose of the device in the past, the additional information of the device includes preliminary estimation information for the camera pose of the device, calculated from the value of an Inertial Measurement Unit (IMU) of the device. can include The preliminary estimation information for the camera pose of the device may include a rough camera pose estimated using IMU information, and the preliminary estimation information may include approximate location information and direction information for the device. When such preliminary estimation information is used, a candidate group of reference images to be matched with the query image may be reduced.

In an embodiment of the present disclosure, the computing device 100 determines candidate reference images to be matched with feature points among a plurality of reference images on a database using additional information of a client device, and the determined candidate reference images. Based on feature point matching between images and the query image, visual localization including pose estimation for a camera of the device may be performed. As such, the additional information of the device may be used to determine a candidate reference image to be matched with a feature point among a plurality of reference images stored in the database. Accordingly, the technique according to an embodiment of the present disclosure may efficiently use computing resources used for feature point matching.

In an embodiment of the present disclosure, the additional information of the device may further include camera information of the device, and an example of such camera information may include an intrinsic parameter of the camera. The computing device 100 may determine a pose estimation algorithm used to perform the visual localization among a plurality of pose estimation algorithms based on the camera information of the device. For example, the pose estimation for a device (eg, camera) may estimate 6DoF in a direction that minimizes a reprojection error.

As an example, if the camera's internal parameters are not given (i.e., if it is an uncalibrated camera), the pose estimation algorithm uses the Direct Linear Transformation (DLT) methodology to determine the elements of the rotation matrix, the translation vector The reprojection error equation including the elements of the vector and the internal parameters of the camera as unknowns can be changed into a linear equation, and the rotation matrix, the translation vector, and the camera matrix can be estimated through QR decomposition.

As an example, given the camera's internal parameters (i.e., a calibrated camera), the pose estimation algorithm uses the Perspective-n-Point (PnP) methodology, which is different from DLT, which transforms the non-linear reprojection error into a linear equation. Preferably, pose estimation can be performed by directly solving the reprojection error, which is a nonlinear equation, using the Gauss-Newton methodology or the Levenberg-Marquardt methodology. In this example, representative PnP methodologies may include P3P, EPnP, and SQPnP methodologies.

As a further example, the above-described DLT and PnP methodologies may introduce RANSAC in order to minimize camera pose errors due to inaccuracies of 3D coordinates of reference feature points or inaccuracies of query feature points.

Therefore, as described above, the technique according to an embodiment of the present disclosure may apply a camera pose estimation algorithm differently depending on whether camera information exists, for example, DLT or PnP depending on whether camera information exists or not. By using the methodology, camera pose estimation can be performed.

In an embodiment, the computing device 100 may extract first feature points of a query image from the query image using an artificial intelligence-based feature point acquisition model. (320).

The computing device 100 may obtain at least one feature point and/or at least one descriptor corresponding to each of the at least one feature point from a query image by using a pre-learned feature point acquisition model. For example, feature points may be coordinates for each feature or points on a query image. For example, the descriptor may include at least one of information about a direction, a size, and/or a relation between neighboring pixels of each of the feature points.

In one embodiment, the feature point acquisition model includes a pre-learned model based on supervised learning to recognize as a feature point a point in which a change in at least one of a color or a geometric pattern of an object in an image exceeds a predetermined threshold. can do. For example, the feature point acquisition model may correspond to a pretrained model using an image-based neural network. As another example, the feature point acquisition model may correspond to a pretrained model using a Transformer-based neural network.

In an embodiment, the computing device 100 may obtain a first detection result corresponding to a predetermined dynamic object in the query image from the query image by using an artificial intelligence-based dynamic object detection model (330). .

In one embodiment, the dynamic object detection model may correspond to an artificial intelligence-based model for detecting a predetermined dynamic object on an image. For example, the dynamic object detection model may include a model pre-trained through supervised learning to receive a query image and detect movable objects (eg, cars, people, etc.) that are not fixed within the query image. For example, the dynamic object detection model may receive a query image and output segmentation results for predetermined movable objects (eg, cars, people, etc.) in the query image.

In one embodiment, the first detection result may include a display result of a region of a non-fixed movable object (eg, a car, a tree, etc.) within the query image.

In one embodiment, the dynamic object detection model, as described below, may be used to remove feature points included in the detected dynamic object and/or may be used to remove text included in the detected dynamic object. When a dynamic object detection model is used, since the number of candidate reference images used to perform matching between feature points of a query image and feature points of a reference image can be reduced, visual localization on an augmented reality and/or virtual reality platform can be achieved. It can be implemented more efficiently.

In an embodiment, the computing device 100 may recognize first text present in the query image from the query image by using a text recognition model. The based text recognition model may, for example, perform an OCR (Optical Character Recognition) function. For example, the text recognition model may recognize an area of text on an input query image and recognize what the text is. As an example, the text recognition model may include an artificial intelligence-based pre-trained model, and examples of such text recognition models include an RNN-based model, a Transformer-based model, and a Bert-based model. can The text recognition model may recognize, for example, car number and/or name of a sign on a query image.

In one embodiment, the computing device 100 may perform a first-second text removed from the query image among the first text and the query image based on a comparison result of the first detection result and the recognized first text. 1-1 text maintained on the above may be determined. The computing device 100 may combine the output of the dynamic object detection model and the output of the text recognition model to remove text belonging to the dynamic object area on the query image. That is, the computing device 100 may determine the first text included in the detection area corresponding to the predetermined dynamic object included in the first detection result as the first-second text to be removed from the query image. Accordingly, the computing device 100 may achieve more efficient comparison and matching between the reference image and the query image by removing information on a dynamic object that is unlikely to be included in the reference image.

In one embodiment, the computing device 100 may detect a predetermined first landmark present in the query image from the query image by using an artificial intelligence-based landmark detection model. A landmark in the present disclosure may mean a specific object that can represent a specific area and/or location. For example, such landmarks may include buildings with high external discrimination, such as Namsan Tower, Lotte Tower, and/or Statue of Liberty. For example, a landmark may be selected among fixed objects.

Landmark detection may correspond to an artificial intelligence-based model that detects a predetermined landmark object on an image. For example, the landmark detection model may include a pretrained model through supervised learning to receive a query image and detect an object corresponding to a predetermined landmark in the query image. For example, the landmark detection model may receive a query image and output segmentation results for predetermined landmark objects in the query image.

In one embodiment, a landmark detection model may be used to efficiently perform visual localization. The landmark detection model receiving the query image may determine whether a specific landmark exists in the query image. When a landmark detection model detects a specific landmark, identification information on the corresponding landmark may be stored as metadata of the corresponding query image. Such metadata may be compared with metadata mapped to a reference image and used to reduce the number of candidate reference images to be compared with a query image among reference images. That is, candidate reference images including the corresponding landmark as metadata may be determined as reference images to be compared with the query image.

In one embodiment, the computing device 100 may perform visual processing of the device capturing the query image, based at least in part on the 1-1 feature points on the query image, the additional information of the device, and the first landmark. localization can be performed. For example, the computing device 100 determines candidate reference images to be matched with feature points among a plurality of reference images on the database using the additional information of the device and the first landmark, and determines the candidate reference images and the determined candidate reference images. Based on feature point matching between the query images, visual localization including pose estimation for a camera of the device may be performed.

In one embodiment, the computing device 100 removes from the query image among the first feature points based on a comparison result between the first feature points obtained from the feature point acquisition model and the first detection result obtained from the dynamic object detection model. 1-2 feature points and 1-1 feature points maintained on the query image may be determined (340). The computing device 100 may perform visual localization on the device that captured the query image based at least in part on the 1-1 feature points on the query image and the additional information of the device.

In an embodiment, the computing device 100 may remove feature points within a region corresponding to a dynamic object obtained from a dynamic object detection model from among feature points within a query image obtained from a feature point acquisition model. For example, the computing device 100 determines first feature points included in a detection area corresponding to a predetermined dynamic object included in the first detection result as 1-2 feature points to be removed from the query image, and First feature points not included in a detection area corresponding to the predetermined dynamic object included in 1 detection result may be determined as 1-1 feature points maintained on the query image. Accordingly, feature point comparison between the reference image and the query image can be performed more efficiently.

In one embodiment, when the computing device 100 performs visual localization on a device that has captured a query image, the reference image stored in the database of the computing device 100, 1-1 feature points on the query image, and Based on the additional information of the device, visual localization may be performed on the device that captured the query image.

In one embodiment, the computing device 100, when performing visual localization on a device that has captured a query image, selects first feature points of the query image and second feature points of the reference image among a plurality of reference images. Relative camera pose information for the query image may be determined according to a result of comparison between 3D camera coordinates allocated to the pose reference image and 2D coordinates of 1-1 feature points of the query image. there is. A pose reference image in the present disclosure may refer to a reference image to be matched with a query image among reference images. A pose reference image in the present disclosure may mean a reference candidate image to be matched with a query image among reference candidate images. As an example, the pose reference image may refer to an image having the highest matching rate with a query image among a plurality of candidate images. In this example, the computing device 100 may perform visual localization on the query image using the 3D camera coordinates assigned to the pose reference image.

In one embodiment, the computing device 100, when performing visual localization on a device that has captured a query image, based on camera pose information (ie, 3D camera coordinates) assigned to a pose reference image and relative camera pose information Thus, absolute camera pose information of the query image with respect to the origin may be determined, and visual localization may be performed for the device that captured the query image based on the absolute camera pose information.

Absolute camera pose information in the present disclosure, for example, a geodetic datum or a geodetic system (or a geodetic reference datum, a geodetic reference system, or a geodetic reference frame) commonly used in a specific region or a specific country, a spatial reference system (SRS) ) or coordinate reference system (CRS), or geographic coordinate system (GCS), and/or coordinate values and/or direction values. Relative camera pose information in the present disclosure may include camera pose information generated by a comparison result between 3D coordinates of camera pose information assigned to a reference image and 2D coordinates of 1-1 feature points of a query image.

In one embodiment, when the computing device 100 obtains feature points and extracts the first feature points for the query image from the query image, pixel coordinates and descriptors corresponding to the first feature points It may include obtaining. The computing device 100 compares descriptors corresponding to the first feature points of the query image with descriptors corresponding to the second feature points of the reference image stored in the database, thereby performing visual localization on the device that captured the query image can do. Since the descriptors are mapped to the query image, they may be compared with descriptors mapped to the reference image and used to determine a candidate reference image to be compared with the query image among a plurality of reference images. Descriptors can be used to efficiently perform a comparison of a reference image and a query image. That is, the computing device 100 compares the descriptor corresponding to the first feature points of the query image with the descriptor corresponding to the second feature points of the reference image stored in the database, so that the visual representation of the device among a plurality of reference images Determining a pose reference image for localization, and corresponding to 3D coordinate information mapped to the pose reference image, matching information between the first feature points and the second feature points, and the first feature points of the query image Based on the pixel coordinates, visual localization including camera pose estimation for the device may be performed.

In an additional embodiment, the computing device 100 compares a descriptor and pixel coordinates corresponding to the first feature points of a query image with a descriptor and pixel coordinates corresponding to second feature points of a reference image stored in a database, thereby providing a plurality of Determine a pose reference image for visual localization for the device among reference images, and 3D coordinate information mapped to the pose reference image, matching information between the first feature points and the second feature points, and the query image Based on pixel coordinates corresponding to the first feature points of , visual localization including camera pose estimation for the device may be performed.

As illustrated in FIG. 3 , computing device 100 may perform pose estimation for a query image captured by a client's device using the pre-trained model(s) for performing visual localization. Operations of the computing device 100 in FIG. 3 exemplarily represent an inference process of a model.

4 exemplarily illustrates a method of storing metadata mapped to a reference image in order to perform visual localization according to an embodiment of the present disclosure.

Steps performed in FIG. 4 may be performed by, for example, the computing device 100 . The operations illustrated in FIG. 4 exemplarily represent a method for generating metadata, a method for constructing a model, and/or a learning method for metadata of the computing device 100 .

In one embodiment, the computing device 100 extracts second feature points for the reference image from the reference image using the feature point acquisition model, and uses the dynamic object detection model to extract the second feature points from the reference image to the reference image. A second detection result corresponding to the determined dynamic object is obtained, and based on a comparison result between the second detection result and the second feature points, 2-2 feature points removed from the reference image among the second feature points and the reference image 2-1 feature points maintained on the image may be determined, and 3-dimensional camera coordinates corresponding to the 2-1 feature points may be acquired from a predefined 3-dimensional map.

As shown in FIG. 4 , the computing device 100 may obtain camera pose information including a reference image and a position and attitude of a camera that has captured the reference image (410). In order to implement an augmented reality or virtual reality platform in a specific region, a plurality of reference images may be captured by a device in the region. Metadata for each of the reference images may be generated and stored in the storage unit of the computing device 100 together with the reference image or feature points of the reference image. For example, the camera pose information may include absolute camera pose information based on an absolute coordinate system on the earth of a device (eg, a camera) that has taken a reference image. Absolute camera pose information based on the absolute coordinate system, for example, geodetic datum or geodetic system (or geodetic reference datum, geodetic reference system, or geodetic reference frame), spatial reference system (SRS) Alternatively, coordinate values and/or direction values for a coordinate reference system (CRS), or a geographic coordinate system (GCS) may be included. As another example, the absolute camera pose information based on the absolute coordinate system is information corresponding to the global coordinate system, such as "latitude, longitude, azimuth", "coordinate system used in a specific region or country", and "items included in GNSS". can mean

In an embodiment, the computing device 100 may extract second feature points for a reference image from the reference image using an artificial intelligence-based feature point acquisition model (420). Here, the feature point acquisition model may correspond to the feature point acquisition model described above with reference to FIG. 3 . For example, each of the second feature points output from the feature point acquisition model may include pixel coordinates and descriptors corresponding to the reference image.

In an embodiment, the computing device 100 may obtain a second detection result corresponding to a predetermined dynamic object in the reference image from the reference image by using an artificial intelligence-based dynamic object detection model (430). . Here, the dynamic object detection model may correspond to the dynamic object detection model described above with reference to FIG. 3 .

The second feature points in FIG. 4 represent feature points corresponding to the reference image, and the first feature points in FIG. 3 indicate feature points corresponding to the query image. Similarly, the second detection result in FIG. 4 represents the output from the dynamic object detection model corresponding to the reference image and the first detection result in FIG. 3 represents the output from the dynamic object detection model corresponding to the query image.

The computing device 100 determines 2-2 feature points removed from the reference image among the second feature points and 2nd feature points maintained on the reference image, based on the comparison result between the second detection result and the second feature points. -1 feature points may be determined (440).

The computing device 100 determines second feature points included in a detection area corresponding to a predetermined dynamic object included in the second detection result as 2-2 feature points to be removed from the query image, and determines the second feature points included in the second detection result. Second feature points not included in a detection area corresponding to the predetermined dynamic object included in may be determined as 2-1 feature points maintained on the query image. When storing the feature points of the reference image as metadata, the computing device 100 may remove feature points included in the dynamic object and store feature points not included in the dynamic object. Accordingly, resource-efficient comparison between the first feature points of the query image and the second feature points of the reference image can be made.

In an embodiment, the computing device 100 may recognize the second text present in the reference image from the reference image by using the text recognition model. The computing device 100 determines the 2-2 text removed from the reference image among the second text and the second text maintained on the reference image, based on the comparison result of the second detection result and the recognized second text. -1 Text can be determined.

The text recognition model in FIG. 4 may correspond to the text recognition model described in FIG. 3 . The second text, the 2-2 text, and the 2-1 text in FIG. 4 are texts corresponding to the reference image, and the first text, the 2-2 text, and the 2-1 text corresponding to the query image in FIG. Can be used to differentiate from text.

When the computing device 100 determines the 2-2 text removed from the reference image and the 2-1 text maintained on the reference image among the second texts, the detection corresponding to the predetermined dynamic object included in the second detection result is performed. The second text included in the region may be determined as the 2-2 text to be removed from the reference image. For example, the coordinate information corresponding to the 2-1 feature points includes 3D camera coordinates corresponding to the 2-1 feature points remaining after the 2-2 feature points are removed on the predefined 3D map. can do. Accordingly, visual localization may be performed according to comparison between the 3D camera coordinates of the feature point corresponding to the reference image stored in the storage unit of the computing device 100 and the feature point (2D coordinates) corresponding to the query image.

The computing device 100 may include the 2-1st text from which the 2-2nd text is removed from among the 2nd text, as the metadata mapped to the reference image. When a model for visual localization is built through the metadata mapped to such a reference image, in the inference process, according to the comparison between the metadata mapped to the query image and the metadata mapped to the reference image, the query image can be more quickly and more resource-efficient visual localization can be implemented.

In an embodiment, the computing device 100 may detect a predetermined second landmark existing in the reference image from the reference image by using an artificial intelligence-based landmark detection model. The landmark detection model in FIG. 4 may correspond to the landmark detection model in FIG. 3 . The second landmark in FIG. 4 is related to the reference image, and the first landmark in FIG. 3 is related to the query image. In one embodiment, the metadata mapped to the reference image may include the second landmark detected by the landmark detection model. When a model for visual localization is built through the metadata mapped to such a reference image, in the inference process, according to the comparison between the metadata mapped to the query image and the metadata mapped to the reference image, the query image can be more quickly and more resource-efficient visual localization can be implemented.

In one embodiment, the computing device 100 may store coordinate information corresponding to the 2-1 feature points, descriptor information corresponding to the 2-1 feature points, and the camera pose information as metadata mapped to a reference image. Yes (450).

In one embodiment, the computing device 100 performs a 3D map corresponding to 2-1 feature points of a reference image in a 3D map (eg, a point cloud or a mesh) prepared by scanning in advance. Camera coordinates can be obtained. These 3D camera coordinates may also be stored as metadata corresponding to the reference image. Such metadata may be compared with the 2D coordinates of feature points of the query image and used to determine relative pose information with respect to the query image.

In one embodiment, the metadata for one reference image includes 2-dimensional pixel coordinates of the 2-1 feature points, descriptors of the 2-1 feature points, 3-dimensional camera coordinates corresponding to the 2-1 feature points, and reference image It may include at least one of camera pose information and GPS information acquired from a photographing device of the camera, landmark information corresponding to a reference image, and/or text information excluding dynamic objects.

In one embodiment, the metadata stored in the database of the computing device 100 may consist of a set of metadata assigned to each reference image unit.

In an additional embodiment of the present disclosure, metadata mapped to a query image includes: pixel coordinates and descriptors corresponding to 1-1 feature points that are maintained on the query image without being removed among the first feature points of the query image; , At least one of location information of the device that captured the query image, a first landmark and a first text corresponding to the query image, and/or camera pose information of the device calculated from an IMU value of the device. there is. Therefore, efficient visual localization of the query image can be performed according to the comparison between the metadata mapped to the query image and the metadata mapped to the reference image.

In a further embodiment of the present disclosure, the query image and the reference image may further include information related to a depth map. By additionally utilizing information about the depth map, the computing device 100 may implement more accurate visual localization.

5 illustratively illustrates the operation of the feature point acquisition model 500 according to an embodiment of the present disclosure.

The description of the feature point acquisition model has been described in detail in FIGS. 3 and 4 , and portions overlapping with those described in FIGS. 3 and 4 will be omitted from the description of FIG. 5 .

In one embodiment, the feature point acquisition model 500 may be a neural network built through deep learning or machine learning. The feature point acquisition model 500 may acquire feature points from an image (eg, a query image and/or a reference image) included in the input data. In addition, the feature point acquisition model 500 may be pre-trained using a dataset. For example, the feature point acquisition model 500 may be trained so that feature points acquired from each of the images correspond to each other when images obtained by photographing regions corresponding to each other in different environments are input.

A dataset may refer to a set of data for learning and verifying a neural network. A dataset may include a training dataset and/or a validation dataset. A training dataset may be a set of data used in a neural network learning process. For example, the training dataset may be a set of data used for learning in the learning process of the feature point acquisition model 500 . A validation dataset may be a set of data used to evaluate a neural network. For example, the verification dataset may be a set of data used to evaluate the feature point acquisition model 500 . In one embodiment, landmark detection models, dynamic object detection models, and/or text recognition models described below may also utilize the aforementioned datasets.

As illustrated in FIG. 5 , the feature point acquisition model 500 may generate an output image 510 in response to an input image 200 . The following input image 200 may correspond to a query image in an inference process, and may correspond to a reference image in a learning process or a DB construction process. The input image 200 may include an image obtained by a client device. In a further embodiment, the input image 200 may include depth map information as well as an RGB or grayscale image.

In one embodiment, the output image 510 may include a plurality of keypoints as a result of the keypoint extraction model and descriptors corresponding to each of the keypoints. Feature points on the output image 510 may include, for example, 2D coordinate values. The feature point acquisition model 500 may output inflection points or color change points of objects on the input image 200 as feature points.

6 exemplarily illustrates operations of the dynamic object detection model 600a and the landmark detection model 600b according to an embodiment of the present disclosure.

The description of the dynamic object detection model 600a and the landmark detection model 600b has been described in detail in FIGS. 3 and 4, and portions overlapping with those described in FIGS. 3 and 4 will be omitted from the description of FIG. 6. will be.

In one embodiment, the dynamic object detection model 600a and the landmark detection model 600b may be neural networks built through deep learning or machine learning. The dynamic object detection model 600a and the landmark detection model 600b may obtain feature points from an image (eg, a query image and/or a reference image) included in the input data 200 . Also, the dynamic object detection model 600a and the landmark detection model 600b may be pre-trained using a dataset. For example, the dynamic object detection model 600a and the landmark detection model 600b, when images of regions corresponding to each other in different environments are input, dynamic objects or landmarks obtained from each of the images are input. They can be learned to correspond to each other.

In one embodiment, the dynamic object detection model 600a may detect or segment predetermined movable objects in response to the input image 200 . In an embodiment, the landmark detection model 600b may detect or segment an object corresponding to a predetermined landmark in response to the input image 200 .

In an embodiment, at least one of the dynamic object detection model 600a and the landmark detection model 600b may detect a predetermined object and display a bounding box around the object to distinguish the object.

In one embodiment, at least one of the dynamic object detection model 600a and the landmark detection model 600b may include an artificial intelligence-based model for detecting a plurality of objects in an image.

In an additional embodiment, the dynamic object detection model 600a and the landmark detection model 600b may be integrated and operated as one model.

As shown in FIG. 6 , the input image 200 may correspond to a query image in FIG. 3 or a reference image in FIG. 4 .

The input image 200 may be input to each of the dynamic object detection model 600a and the landmark detection model 600b. In response to the input image 200 , the dynamic object detection model 600a may generate an output image 610 including detection results of dynamic objects. Dynamic objects 610a, 610b, and 610c corresponding to vehicles and dynamic objects 610d corresponding to trees may be included in the output image 610 . Illustrated contents ( 610a , 610b , 610c , and 610d ) of dynamic objects are merely examples, and the dynamic object detection model 600a may perform detection of various types of movable objects. The output 610 of the dynamic object detection model 600a can be used to detect feature points and texts that do not belong to the dynamic object, so that a process of comparing a query image and a reference image can be implemented more efficiently.

In response to the input image 200 , the landmark detection model 600b may generate an output image 620 including detection results of object(s) corresponding to the pre-stored landmark. A landmark 620a corresponding to "Seoul High Court" may be included on the output image 620 . Identification information on the landmark 620a may be stored as metadata for a query image and/or a reference image, so that a process of comparing the query image and the reference image may be implemented more efficiently.

7 exemplarily illustrates an operation of a text recognition model 700 according to an embodiment of the present disclosure.

The description of the text recognition model 700 has been described in detail in FIGS. 3 and 4 , and portions overlapping with those described in FIGS. 3 and 4 will be omitted from the description of FIG. 7 .

The input image 200 may be input to the text recognition model 700 . As described above, the input image 200 may be input to the feature point acquisition model 500, the dynamic object detection model 600a, the landmark detection model 600b, and the text recognition model 700.

The text recognition model 700 may detect regions of presence of text within the input image 200 and/or determine what text within the input image 200 means.

The text recognition model 700 may include any type of model capable of performing Optical Character Recognition (OCR).

The text recognition model 700 may include any type of model for recognizing text based on artificial intelligence. For example, the text recognition model 700 may perform preprocessing to change the brightness and/or color of the input image 200 to facilitate recognition of texts within the input image 200 .

The text recognition model 700 can locate texts and create bounding boxes for these texts. For example, in order to recognize positions of texts on the input image 200, the text recognition model 700 may use a Convolutional Neural Network (CNN) model, which is an image-based deep learning model.

The text recognition model 700 may recognize the content of text within a bounding box corresponding to the position of the text. As an example, the text recognition model 700 may use a Recurrent Neural Network (RNN) series model to recognize text within a bounding box. Additionally, the text recognition model 700 may use a transformer and/or attention-based deep learning model to recognize text within a bounding box.

In the example described above, the text recognition model 700 may include a model in which a first model for finding a location for text and a second model for recognizing the content of text within the location of the text are combined. there is. In a further embodiment, the function of determining the position of the text may be performed by at least one of the aforementioned dynamic object detection model 600a and landmark detection model 600b.

As shown in FIG. 7 , the output 710 of the text recognition model 700 may be represented by examples such as signboards 710a and 710b of a building and number 710c of a vehicle. As such, the text recognition model 700 may display a location area for text as a bounding box and generate an output 710 including recognition information on text. Recognition information on the text illustrated in FIG. 7 may include “Myeongsan Building” 710a, “Personal bankruptcy/rehabilitation corporate bankruptcy/rehabilitation” 710b, and “13bo 6436” 710c.

In an embodiment, the text recognition model 700 may recognize text expressed vertically or horizontally in units of keywords or sentences. In this example, the text recognition model 700 does not recognize the text "Myeongsan Building" corresponding to reference number 710a as "Myeong", "Mountain", "Building", and "Ding", respectively, but the entire word "Myeongsan Building". can be recognized as one.

The location area for the text may be compared with the output of the dynamic object detection model 600a and divided into text included in the dynamic object and text not included in the dynamic object. Text included in the dynamic object will be removed on the image and text not included in the dynamic object may be kept on the image. In addition, text recognition information may be mapped to the input image 200 (eg, a query image or a reference image) and stored. Recognition information on text may be used as reference information in performing comparison between a query image and a reference image. Accordingly, as reference images having text information corresponding to text information included in the query image are determined as candidate reference images, computing resources used for comparison between the query image and the reference image may be reduced.

8 illustratively illustrates a flow of image processing for efficiently performing visual localization according to an embodiment of the present disclosure.

The image processing illustrated in FIG. 8 may be performed by, for example, the computing device 100 .

As shown in FIG. 8 , an output image 710 of the text recognition model 700 and an output image 610 of the dynamic object detection model 600a may be compared with each other. For example, a comparison may be made between regions (eg, segmented regions) detected in the output image 710 of the text recognition model 700 and the output image 610 of the dynamic object detection model 600a. For example, objects in the output image 710 of the text recognition model 700 and the output image 610 of the dynamic object detection model 600a may be compared. According to this comparison, the output image 710 of the text recognition model 700 corresponding to the regions 610a, 610b, 610c, and 610d detected as dynamic objects on the output image 610 of the dynamic object detection model 600a The text areas 710c of may be determined. In this example, the text area corresponding to reference number 710c in the output image 710 of the text recognition model 700 is determined as a text area included in the dynamic object, and the text areas corresponding to reference numbers 710a and 710b are dynamic. It may be determined as text areas not included in the object.

In an embodiment, in a process of comparing objects in the output image 710 of the text recognition model 700 and the output image 610 of the dynamic object detection model 600a, the intermediate output image corresponding to reference number 810 810 may be created. On the intermediate output image 810, boundary regions 710a, 710b, and 710c corresponding to text and boundary regions 610a, 610b, 610c, and 610d for dynamic objects may be integrated. Objects in the output image 710 of the text recognition model 700 and the output image 610 of the dynamic object detection model 600a may be compared on the intermediate output image 810 . In one embodiment, such an intermediate output image 810 is an exemplary image for convenience of explanation, and an output image 820 corresponding to reference numeral 820 is provided without generating the intermediate output image 810 according to an implementation aspect. may be created.

In one embodiment, according to the comparison of the objects in the output image 710 of the text recognition model 700 and the output image 610 of the dynamic object detection model 600a, the text 710c corresponding to the dynamic object An output image or an output result 820 may be generated in which is removed and the texts 710a and 710b that do not correspond to the dynamic object are maintained. In one embodiment, based on a comparison between these text areas and dynamic object areas on a query image or reference image, the computing device 100 may efficiently construct reference images to be compared in performing visual localization. And effective comparison between the query image and the reference image can be made in performing visual localization.

As such, since the image processing technique according to an embodiment of the present disclosure can efficiently and selectively store text information stored as metadata, efficient management of metadata is possible. Furthermore, in the image processing technique according to an embodiment of the present disclosure, as metadata is stored as text information on fixed objects, metadata extracted from a query image input through a virtual reality and/or augmented reality platform and The number of reference images to be compared with a query image may be reduced through a method of comparing metadata corresponding to a reference image.

9 illustratively illustrates a flow of image processing for efficiently performing visual localization according to an embodiment of the present disclosure.

Image processing illustrated in FIG. 9 may be performed by, for example, the computing device 100 .

As shown in FIG. 8 , output information 510 of the feature point acquisition model 500 and output information 610 of the dynamic object detection model 600a may be compared with each other. For example, comparison may be made between regions (eg, segmented regions) detected in the output information 510 of the feature point acquisition model 500 and the output information 610 of the dynamic object detection model 600a. For example, a comparison may be made to determine whether regions in the output information 510 of the feature point acquisition model 500 and the output information 610 of the dynamic object detection model 600a correspond to each other. Comparison of determining whether areas in the output information 510 of the feature point acquisition model 500 and the output information 610 of the dynamic object detection model 600a correspond to each other is exemplarily shown in reference numeral 910 of FIG. 9 . is shown According to this comparison, the output information 510 of the feature point acquisition model 500 corresponding to the regions 610a, 610b, 610c, and 610d detected as dynamic objects on the output information 610 of the dynamic object detection model 600a The feature points of can be determined. In this example, the feature points included in reference numbers 610a, 610b, 610c, and 610d of the dynamic object detection model 600a among the output information 510 of the feature point acquisition model 500 are determined as feature points included in the dynamic object, Also, feature points not included in reference numerals 610a, 610b, 610c, and 610d may be determined as feature points not included in the dynamic object.

For example, when the degree of correspondence between the location information of feature points and the area information of a dynamic object exceeds a predetermined threshold correspondence value or the distance between the location information of feature points and the area information of a dynamic object is less than a predetermined threshold distance value, the corresponding feature points are It may be determined as feature points included in the dynamic object.

As shown by reference number 920, among the output information 510 of the feature point acquisition model 500, feature points included in reference numbers 610a, 610b, 610c, and 610d of the dynamic object detection model 600a are feature points included in the dynamic object. Since it is determined as , corresponding feature points may be removed from the output image 920 . As shown by reference number 920, among the output information 510 of the feature point acquisition model 500, feature points not included in reference numbers 610a, 610b, 610c, and 610d of the dynamic object detection model 600a are not included in the dynamic object. Since the feature points are determined as unrelated feature points, the corresponding feature points may be maintained on the output image 920 .

In one embodiment, based on the comparison between these feature points and the dynamic object regions on the query image or reference image, the computing device 100 may efficiently construct reference images to be compared in performing visual localization. And effective comparison between the query image and the reference image can be made in performing visual localization.

As such, since the image processing technique according to an embodiment of the present disclosure can efficiently and selectively store feature point information stored as metadata, efficient management of metadata is possible. Furthermore, in the image processing technique according to an embodiment of the present disclosure, as metadata is stored as feature point information on fixed objects, metadata extracted from a query image input through a virtual reality and/or augmented reality platform and The number of reference images to be compared with a query image may be reduced through a method of comparing metadata corresponding to a reference image.

10 illustratively illustrates an operation of a model 1000 for performing visual localization according to an embodiment of the present disclosure.

In one embodiment, the model 1000 for performing visual localization may be included in the computing device 100 and executed by a processor of the computing device 100 .

An input image 200 may be input to the model 1000 . The input image 200 may be input to the feature point acquisition model 500, the dynamic object detection model 600a, the landmark detection model 600b, and the text recognition model 700.

A first output 510 of the feature point acquisition model 500 and a second output 610 of the dynamic object detection model 600a may be compared. According to this comparison, the model 1000 of the computing device 100 may generate a sixth output 920 . The sixth output 920 may include information about feature points not included in the dynamic object. Through the sixth output 920, the computing device 100 may selectively store feature points not included in the dynamic object, and thus perform comparison between feature points between a query image and a reference image more resource-efficiently.

A fourth output 710 of the text recognition model 700 and a second output 610 of the dynamic object detection model 600a may be compared. According to this comparison, the model 1000 of the computing device 100 may generate a fifth output 820 . The fifth output 820 may include information about texts not included in the dynamic object. Through the fifth output 820, the computing device 100 may selectively store texts not included in the dynamic object. Since the computing device 100 may select only some of the reference images among all reference images as reference images to be compared with the query image by using a method of comparing text information between the query image and the reference image, visual localization is more resource efficient. - Can be done efficiently.

A third output 620 of the landmark detection model 600b may be output by the model 1000 of the computing device 100 . The third output 620 includes information corresponding to landmarks and may be used as metadata for reference data and metadata for query data. When a query image including a specific landmark is input, the computing device 100 may select reference image(s) including the specific landmark as a reference image to be compared with the query image.

As shown in FIG. 10 , the visual localization technique according to an embodiment of the present disclosure may use at least one of a third output 620 , a fifth output 820 , and a sixth output 920 . As at least one of the outputs 620 , 820 , and 920 is used as metadata, the computing device 100 can effectively select a reference image to be compared with the query image without comparing the entire reference image with the query image.

In one embodiment, in the process of storing the reference image in the database, 2D pixel coordinate values of feature points included in the sixth output 920 and/or values of descriptors of the feature points may be stored as metadata. In addition, in the process of storing the reference image in the database, the 3D camera coordinates of the feature points included in the sixth output 920 on the 3D map (eg, point cloud format or mesh format) produced by scanning are metadata. can be saved as In addition, in the process of storing the reference image in the database, the relative camera pose or 6DoF value for the absolute coordinate system of the 3D map of the client's device (e.g., camera) taking the reference image, and/or the GPS value are metadata. can be stored Also, information corresponding to the landmark included in the third output 620 may be stored as metadata. Also, text information not included in the dynamic object included in the fifth output 820 may be stored as metadata. In addition, a camera pose and GPS information of a reference image corresponding to the input image 200 may be stored as metadata.

In one embodiment, in the process of receiving and processing a query image, 2D pixel coordinate values of feature points included in the sixth output 920 and/or values of descriptors of the feature points may be stored as metadata. In addition, in the process of receiving and processing the query image, information corresponding to the landmark included in the third output 620 may be stored as metadata. In addition, in the process of receiving and processing the query image, text information not included in the dynamic object included in the fifth output 820 may be stored as metadata. In addition, the camera pose and GPS information of the query image corresponding to the input image 200 may be stored as metadata. In addition, preliminary estimation information of a query image corresponding to the input image 200 (eg, preliminary estimation information about a camera pose of a query image estimated based on IMU information of a device that has captured the query image) is stored as metadata. can At least some of the metadata of the query image may be used to quickly reduce a candidate group of reference images in a database for feature point matching. When performing feature point matching between candidate groups of reference images and a query image, pixel coordinates and descriptors of feature points extracted from two images may be used. 3D coordinate values of the feature points of the reference image that matched the query image best (ie, the highest degree of correspondence), matching information between feature points, pixel coordinate values of the feature points of the query image, etc. are used to obtain the query image. Camera pose estimation may be performed for

11 depicts a simplified and general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

A component, module or unit in this disclosure includes routines, procedures, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, those skilled in the art will understand that the methods presented in this disclosure can be used in single-processor or multiprocessor computing devices, minicomputers, mainframe computers as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like ( It will be fully appreciated that each of these may be implemented with other computer system configurations, including those that may be operative in connection with one or more associated devices.

Embodiments described in this disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

A computing device typically includes a variety of computer readable media. Computer readable media can be any medium that can be accessed by a computer, including volatile and nonvolatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer readable media may include computer readable storage media and computer readable transmission media.

Computer readable storage media are volatile and nonvolatile media, transitory and non-transitory, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. includes media Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device. device, or any other medium that can be accessed by a computer and used to store desired information.

A computer readable transmission medium typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism. Including all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information within the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable transmission media.

An exemplary environment 2000 implementing various aspects of the present invention is shown comprising a computer 2002, which includes a processing unit 2004, a system memory 2006 and a system bus 2008. do. Computer 200 herein may be used interchangeably with a computing device. System bus 2008 couples system components, including but not limited to system memory 2006, to processing unit 2004. Processing unit 2004 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as the processing unit 2004.

System bus 2008 can be any of several types of bus structures that can additionally be interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 2006 includes read only memory (ROM) 2010 and random access memory (RAM) 2012 . The basic input/output system (BIOS) is stored in non-volatile memory 2010, such as ROM, EPROM, EEPROM, etc. BIOS is a basic set of information that helps transfer information between components within the computer 2002, such as during startup. contains routines. RAM 2012 may also include high-speed RAM, such as static RAM, for caching data.

The computer 2002 may also read from an internal hard disk drive (HDD) 2014 (eg EIDE, SATA), a magnetic floppy disk drive (FDD) 2016 (eg a removable diskette 2018), or for writing to them), SSDs and optical disk drives 2020 (for example, for reading CD-ROM disks 2022 or reading from or writing to other high capacity optical media such as DVDs) include The hard disk drive 2014, magnetic disk drive 2016, and optical disk drive 2020 are connected to the system bus 2008 by the hard disk drive interface 2024, magnetic disk drive interface 2026, and optical drive interface 2028, respectively. ) can be connected to The interface 2024 for external drive implementation includes, for example, at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide non-volatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 2002, drives and media correspond to storing any data in a suitable digital format. Although the description of computer-readable storage media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art can use zip drives, magnetic cassettes, flash memory cards, cartridges, It will be appreciated that other types of computer-readable storage media, such as those of other types, may also be used in the exemplary operating environment and that any such media may contain computer-executable instructions for performing the methods of the present invention. .

A number of program modules may be stored on the drive and RAM 2012, including an operating system 2030, one or more application programs 2032, other program modules 2034, and program data 2036. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 2012. It will be appreciated that the present invention may be implemented in a variety of commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 2002 through one or more wired/wireless input devices, such as a keyboard 2038 and a pointing device such as a mouse 2040. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. Although these and other input devices are often connected to the processing unit 2004 through an input device interface 2042 that is connected to the system bus 2008, a parallel port, IEEE 1394 serial port, game port, USB port, IR interface, may be connected by other interfaces such as the like.

A monitor 2044 or other type of display device is also connected to the system bus 2008 through an interface such as a video adapter 2046. In addition to the monitor 2044, computers typically include other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 2002 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 2048 via wired and/or wireless communications. Remote computer(s) 2048 may be workstations, server computers, routers, personal computers, handheld computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, and generally relate to computer 2002. Although many or all of the described components are included, for brevity, only memory storage device 2050 is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) 2052 and/or a larger network, such as a wide area network (WAN) 2054 . Such LAN and WAN networking environments are common in offices and corporations and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to worldwide computer networks, such as the Internet.

When used in a LAN networking environment, computer 2002 is connected to local network 2052 through wired and/or wireless communication network interfaces or adapters 2056. Adapter 2056 may facilitate wired or wireless communications to LAN 2052, which also includes a wireless access point installed therein to communicate with wireless adapter 2056. When used in a WAN networking environment, computer 2002 may include a modem 2058, be connected to a communications server on WAN 2054, or other means of establishing communications over WAN 2054, such as over the Internet. have Modem 2058, which can be internal or external and wired or wireless, is connected to system bus 2008 through serial port interface 2042. In a networked environment, program modules described for computer 2002, or portions thereof, may be stored in remote memory/storage device 2050. It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between computers may be used.

Computer 1602 is any wireless device or entity that is deployed and operating in wireless communication, such as printers, scanners, desktop and/or portable computers, portable data assistants (PDAs), communication satellites, and associated with wireless detectable tags. It operates to communicate with arbitrary equipment or places and telephones. This includes at least Wi-Fi and Bluetooth wireless technologies. Thus, the communication may be a predefined structure as in conventional networks or simply an ad hoc communication between at least two devices.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of example approaches. Based upon design priorities, it is to be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure. The method claims of this disclosure present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

Claims (16)

A method performed by a computing device, comprising:
Receiving a query image and additional information of the device from a device of a client;
acquiring first keypoints of the query image from the query image by using an artificial intelligence-based keypoints acquisition model receiving the query image;
obtaining a first detection result corresponding to a predetermined dynamic object in the query image from the query image by using an artificial intelligence-based dynamic object detection model receiving the query image;
recognizing first text existing in the query image from the query image by using a text recognition model that receives the query image;
Based on a first comparison result between the predetermined dynamic object included in the first detection result and the first feature points and a second comparison result between the predetermined dynamic object included in the first detection result and the first text 1-2 feature points removed from the query image among the first feature points and 1-1 feature points maintained on the query image are determined, and 1-2 feature points removed from the query image among the first texts are determined. 2. Determining text and 1-1st text retained on the query image - 1-2nd text from which text included in a detection area corresponding to the predetermined dynamic object among the first texts is removed from the query image determined by -; and
At least one candidate reference image to be compared with the query image is determined by comparing 1-1 text on the query image with text information allocated as metadata to the reference image, and on the determined candidate reference image, the query image performing visual localization on the device that captured the query image, based at least in part on 1-1 feature points of and additional information of the device;
including,
method. According to claim 1,
The additional information of the device includes location information of the device, and
If a record of estimating the past camera pose of the device exists, the additional information of the device includes preliminary estimation information for the camera pose of the device calculated from the value of an inertial measurement unit (IMU) of the device ,
method. According to claim 2,
The step of performing visual localization on the device that captured the query image is:
determining candidate reference images to be matched with feature points from among a plurality of reference images in a database by using the additional information of the device; and
performing visual localization including pose estimation for a camera of the device based on feature point matching between the determined candidate reference images and the query image;
including,
method. According to claim 2,
The additional information of the device further includes camera information of the device, and
Further based on the camera information of the device, a pose estimation algorithm used to perform the visual localization among a plurality of pose estimation algorithms is determined.
method. delete delete According to claim 1,
detecting a predetermined first landmark existing in the query image from the query image using an artificial intelligence-based landmark detection model;
It further includes, and
Performing visual localization on the device that has taken the query image,
Performing visual localization on the device that captured the query image based at least in part on the determined candidate reference image, 1-1 feature points on the query image, additional information of the device, and the first landmark Including the steps of
method. According to claim 7,
The step of performing visual localization on the device that captured the query image is:
determining candidate reference images to be matched with feature points among a plurality of reference images on a database, based on the additional information of the device and the first landmark; and
performing visual localization including pose estimation for a camera of the device based on feature point matching between the determined candidate reference images and the query image;
including,
method. delete According to claim 1,
The step of performing visual localization on the device that captured the query image is:
Among a plurality of reference images, 3D camera coordinates assigned to a pose reference image determined based on matching between the first feature points of the query image and the second feature points of the reference image and the 1-1 feature points of the query image determining relative camera pose information for the query image according to a comparison result between 2D coordinates;
determining absolute camera pose information of the query image with respect to an origin based on 3D camera coordinates assigned to the pose reference image and the relative camera pose information; and
performing visual localization on the device that captured the query image based on the absolute camera pose information;
including,
method. According to claim 1,
Extracting first feature points for the query image from the query image,
Acquiring pixel coordinates and descriptors corresponding to the first feature points, and
Performing visual localization on the device that has taken the query image,
Performing visual localization on the device that captured the query image by comparing descriptors corresponding to the first feature points of the query image with descriptors corresponding to second feature points of a reference image stored in a database. doing,
method. According to claim 11,
The step of performing visual localization on the device that captured the query image is:
By comparing the descriptor corresponding to the first feature points of the query image with the descriptor corresponding to the second feature points of a reference image stored in a database, a pose reference image for visual localization for the device among a plurality of reference images deciding; and
Based on 3D coordinate information mapped to the pose reference image, matching information between the first feature points and the second feature points, and pixel coordinates corresponding to the first feature points of the query image, the camera for the device performing visual localization including pose estimation;
including,
method. According to claim 1,
extracting second feature points for the reference image from the reference image using the feature point acquisition model;
obtaining a second detection result corresponding to a predetermined dynamic object in the reference image from the reference image by using the dynamic object detection model; and
Determining 2-2 feature points removed from the reference image and 2-1 feature points maintained on the reference image among the second feature points based on a comparison result between the second detection result and the second feature points doing;
Including more,
method. According to claim 1,
Determining 1-2 feature points removed from the query image and 1-1 feature points maintained on the query image among the first feature points based on a comparison result between the first detection result and the first feature points The steps are,
determining first feature points included in a detection area corresponding to the predetermined dynamic object included in the first detection result as 1-2 feature points to be removed from the query image, and determining first feature points not included in a detection area corresponding to a predetermined dynamic object as 1-1st feature points maintained on the query image;
including,
method. A computer program stored on a computer-readable storage medium, which, when executed by a computing device, causes the computing device to perform the following operations:
receiving a query image and additional information of the device from a device of a client;
extracting first feature points of the query image from the query image by using an artificial intelligence-based feature point acquisition model receiving the query image;
obtaining a first detection result corresponding to a predetermined dynamic object in the query image from the query image by using an artificial intelligence-based dynamic object detection model receiving the query image;
recognizing first text existing in the query image from the query image by using a text recognition model receiving the query image;
Based on a first comparison result between the predetermined dynamic object included in the first detection result and the first feature points and a second comparison result between the predetermined dynamic object included in the first detection result and the first text 1-2 feature points removed from the query image among the first feature points and 1-1 feature points maintained on the query image are determined, and 1-2 feature points removed from the query image among the first texts are determined. 2 Determining text and 1-1 text maintained on the query image - 1-2 text from which text included in a detection area corresponding to the predetermined dynamic object among the first texts is removed from the query image determined by -; and
At least one candidate reference image to be compared with the query image is determined by comparing 1-1 text on the query image with text information allocated as metadata to the reference image, and on the determined candidate reference image, the query image performing visual localization on the device that captured the query image, based at least in part on 1-1 feature points of and additional information of the device;
including,
A computer program stored on a computer readable storage medium. As a computing device,
at least one processor; and
Memory;
Including,
The at least one processor is:
receiving a query image and additional information of the device from a device of a client;
extracting first feature points of the query image from the query image by using an artificial intelligence-based feature point acquisition model receiving the query image;
obtaining a first detection result corresponding to a predetermined dynamic object in the query image from the query image by using an artificial intelligence-based dynamic object detection model receiving the query image;
recognizing first text existing in the query image from the query image by using a text recognition model receiving the query image;
Based on a first comparison result between the predetermined dynamic object included in the first detection result and the first feature points and a second comparison result between the predetermined dynamic object included in the first detection result and the first text 1-2 feature points removed from the query image among the first feature points and 1-1 feature points maintained on the query image are determined, and 1-2 feature points removed from the query image among the first texts are determined. 2 Determining text and 1-1 text maintained on the query image - 1-2 text from which text included in a detection area corresponding to the predetermined dynamic object among the first texts is removed from the query image determined by -; and
At least one candidate reference image to be compared with the query image is determined by comparing 1-1 text on the query image with text information allocated as metadata to the reference image, and on the determined candidate reference image, the query image performing visual localization on the device that captured the query image, based at least in part on 1-1 feature points of and additional information of the device;
to do,
computing device.