KR102600939B1 - Apparatus and method for generating data for visual localization - Google Patents

Abstract

According to an embodiment of the present disclosure, a method for generating data for visual localization performed by a computing device is disclosed. The method includes: obtaining a first image and a first depth map from a photographing device, applying a viewpoint transformation to the first image and the first depth map, thereby producing a transformed Acquiring converted 3D data including an image and a converted depth map, using a pre-trained artificial intelligence-based first feature point acquisition model to obtain first feature points (interest points) corresponding to the converted image. Obtaining, obtaining restored 3D data by applying viewpoint inverse transformation to the converted image including the converted depth map and the first feature points, and the restored 3D data. Based on data, it may include generating labeling information corresponding to the first image.

Description

Method and apparatus for generating data for visual localization {APPARATUS AND METHOD FOR GENERATING DATA FOR VISUAL LOCALIZATION}

This disclosure relates to image processing and more specifically to visual localization.

As the demand for Location Based Services increases, the need for accurate location information has increased. The most common way to determine location on mobile devices and mobile platforms is GNSS (Global Navigation Satellite System). However, because GNSS signals may be blocked by obstacles in an indoor environment, there is a limitation that they can only be easily used in an outdoor environment.

Although various technologies have been proposed to perform location recognition indoors, the reality is that many indoor location recognition techniques do not go beyond the fingerprint printing-based location recognition algorithm using wireless signals. In this method, the collected data related to Wi-Fi RSS (Received Signal Strength) or MFS (Magnetic Field Strength) is compared with data from a fingerprinting database. Fingerprinting-based systems have the advantage of being easy to build, but it can be difficult to maintain good performance because the signal pattern itself is affected by changes in the system environment. To overcome the deficiencies of these fingerprint-based systems, many alternatives have been proposed, including Optical, RFID (Radio Frequency Identification), Bluetooth Beacons, ZigBee, and Pseudo Satellite, but these alternatives also have difficulty achieving high accuracy in complex indoor environments. It is evaluated as difficult.

Recently, research on VPS (Visual Positioning System) has been actively conducted as an alternative to implement highly accurate position estimation even in indoor environments. VPS technology can also be expressed as visual localization technology, which refers to a technology that estimates the current location or pose of a device using images taken indoors or outdoors. Pose estimation of a device (or camera) refers to determining the translation and rotation information of a dynamically changing camera viewpoint. This camera pose estimation technology is being used in a variety of fields such as mixed reality, augmented reality, robot navigating, and 3-Dimensional Scene Reconstruction.

US registered patent US10,977,554

This disclosure was made in response to the above-described background technology, and is intended to efficiently generate learning data in a model for estimating camera pose. The present disclosure is intended to increase the accuracy of camera pose estimation and efficiently use time and/or resources required for camera pose estimation.

The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

According to some embodiments of the present disclosure for solving the problems described above, a method performed by a computing device is disclosed. The method includes: acquiring a first image and a first depth map from an imaging device; Obtaining transformed 3D data including a transformed image and a transformed depth map by applying a viewpoint transformation to the first image and the first depth map; Acquiring first feature points corresponding to the converted image using a pre-trained artificial intelligence-based first feature point acquisition model; Obtaining restored 3D data by applying viewpoint inverse transformation to the converted image including the converted depth map and the first feature points; Based on the restored 3D data, it may include generating labeling information corresponding to the first image.

In one embodiment, obtaining the first image and the first depth map from the photographing device includes obtaining the first image, the first depth map, and camera information of the photographing device from the photographing device. can do. The step of acquiring the converted 3D data including the converted image and the converted depth map is: converting the first image having a pixel coordinate system into a normalized coordinate system using the camera information. converting; Converting the first image having the normalized coordinate system into a three-dimensional camera coordinate system using the obtained depth map; and obtaining the transformed 3D data including the transformed image and the transformed depth map by applying viewpoint transformation to the first image having the camera coordinate system.

In one embodiment, the viewpoint transformation is performed by converting the converted first image to be based on a three-dimensional camera coordinate system indicating the location of the photographing device and the direction in which the photographing device is looking from a first coordinate value on the camera coordinate system. It can be converted to a second coordinate value that is different from the first coordinate value.

In one embodiment, the viewpoint transformation may include a rotation matrix and a translation vector. The viewpoint transformation can be performed using a rotation matrix and translation vector.

In one embodiment, the step of obtaining the transformed 3D data includes applying a plurality of viewpoint transformations in which at least one of the rotation matrix and the translation vector is different from the first image and the first depth map, by applying a plurality of viewpoint transformations to the first image and the first depth map. It may include obtaining a plurality of converted 3D data including converted images and a plurality of converted depth maps. The step of generating labeling information corresponding to the first image includes generating the labeling information corresponding to the first image based on a plurality of restored 3D data corresponding to each of the plurality of converted 3D data. May include steps.

In one embodiment, in the viewpoint transformation, a unique transformation matrix may be applied to each pixel of the first image.

In one embodiment, the first image may be acquired from the photographing device in an unlabeled state. Generating labeling information corresponding to the first image may include generating the labeling information indicating feature points corresponding to the first image.

In one embodiment, the first feature point acquisition model corresponds to a pre-trained model based on supervised learning to recognize as a feature point a point where the amount of change in at least one of color or geometric pattern in the object exceeds a predetermined threshold. It can be.

In one embodiment, the viewpoint inverse transformation includes a rotation matrix (R ^-1 ) corresponding to the inverse of the rotation matrix (R) used in the viewpoint transformation, and a translation vector used in the viewpoint transformation. The translational vector (-R ^-1 * T) corresponding to the inverse of (T) can be used.

In one embodiment, the step of acquiring first feature points corresponding to the converted image includes: using the converted depth map, converting the converted 3D data having a 3D camera coordinate system into a normalized coordinate system to create a 2D image. acquiring data; Obtaining the converted image by converting the two-dimensional data having the normalized coordinate system into a pixel coordinate system based on camera information of the photographing device obtained from the photographing device; and inputting the converted image into the first feature point acquisition model to obtain first feature points corresponding to the converted image.

In one embodiment, the step of acquiring the restored 3D data includes: obtaining the converted image including the first feature points by combining the first feature points with the converted image; Converting the converted image including the first feature points into a normalized coordinate system using camera information of the photographing device obtained from the photographing device; Converting the converted image including the first feature points having the normalized coordinate system into a three-dimensional camera coordinate system using the converted depth map; and obtaining the restored 3D data by applying viewpoint inverse transformation to the converted image including the first feature points having the 3D camera coordinate system.

In one embodiment, based on the reconstructed 3D data, generating labeling information corresponding to the image includes: applying a reconstructed depth map in the reconstructed 3D data to the reconstructed 3D data. , obtaining 2D reconstructed data having a 2D normalized coordinate system from the reconstructed 3D data having a 3D camera coordinate system; and a reconstructed image by converting the two-dimensional reconstructed data into a pixel coordinate system based on applying camera information of the photographing device obtained from the photographing device to the two-dimensional reconstructed data having the two-dimensional normalized coordinate system. Obtaining a; It may include generating labeling information corresponding to the first image based on the reconstructed image having the pixel coordinate system.

In one embodiment, the method uses a first training dataset including the first image and labeling information corresponding to the first image to generate first input data corresponding to a first viewpoint of the photographing device. And it further includes training a second feature point acquisition model to output feature points corresponding to the second input data corresponding to the second viewpoint, where the first viewpoint and the second viewpoint are different from each other. You can have a viewpoint.

In one embodiment, the step of training the second feature point acquisition model includes: generating second training data having the second viewpoint by applying the viewpoint transformation to the first training data having the first viewpoint. steps; Inputting the first learning data into the second feature point acquisition model to obtain second feature points corresponding to the first learning data; acquiring third feature points corresponding to the second learning data by inputting the second learning data into the second feature point acquisition model; Restoring the viewpoint of the second learning data in which the third feature points are reflected to the first viewpoint by applying the viewpoint inverse transformation to the second learning data in which the third feature points are reflected; And it will include the step of training the second feature point acquisition model based on comparing the position information of the third feature points in the second learning data restored to the first viewpoint with the position information of the second feature points. You can.

In one embodiment, the step of training the second feature point acquisition model is such that the position information of the third feature points in the second learning data restored to the first viewpoint is the same as the position information of the second feature points. It can be based on a set loss function.

In one embodiment, a computer program stored on a computer-readable storage medium is disclosed. The computer program, when executed by a computing device, causes the computing device to perform the following operations, which operations include: obtaining a first image and a first depth map from an imaging device; Obtaining transformed 3D data including a transformed image and a transformed depth map by applying viewpoint transformation to the first image and the first depth map; An operation of acquiring first feature points corresponding to the converted image using a pre-trained artificial intelligence-based first feature point acquisition model; Obtaining restored 3D data by applying viewpoint inverse transformation to the converted image including the converted depth map and the first feature points; Based on the restored 3D data, the method may include generating labeling information corresponding to the first image.

In one embodiment, a computing device is disclosed. The computing device includes at least one processor; and memory. The at least one processor is configured to: obtain a first image and a first depth map from an imaging device; Obtaining transformed 3D data including a transformed image and a transformed depth map by applying viewpoint transformation to the first image and the first depth map; An operation of acquiring first feature points corresponding to the converted image using a pre-trained artificial intelligence-based first feature point acquisition model; Obtaining restored 3D data by applying viewpoint inverse transformation to the converted image including the converted depth map and the first feature points; Based on the restored 3D data, an operation of generating labeling information corresponding to the first image may be performed.

A method, apparatus, and computer program according to an embodiment of the present disclosure can efficiently generate training data in a model for estimating a camera pose. A method, device, and computer program according to an embodiment of the present disclosure can increase the accuracy of camera pose estimation and efficiently use time and/or resources required for camera pose estimation.

The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. .

Various aspects will now be described with reference to the drawings, where like reference numerals are used to collectively refer to like elements. In the following examples, for purposes of explanation, numerous specific details are set forth to provide a comprehensive understanding of one or more aspects. However, it will be clear that such aspect(s) may be practiced without these specific details.
1 schematically shows a block diagram of a computing device according to an embodiment of the present disclosure.
Figure 2 is a schematic diagram illustrating a network function according to an embodiment of the present disclosure.
Figure 3 exemplarily shows a method of generating labeling information for an image based on viewpoint transformation according to an embodiment of the present disclosure.
Figure 4 exemplarily shows a method of generating labeling information for an image based on viewpoint transformation according to an embodiment of the present disclosure.
Figure 5 exemplarily illustrates viewpoint conversion according to an embodiment of the present disclosure.
FIG. 6 exemplarily illustrates transformation of coordinate systems based on viewpoint transformation according to an embodiment of the present disclosure.
Figure 7 exemplarily shows a learning method of a second feature point acquisition model according to an embodiment of the present disclosure.
8 shows a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the disclosure. However, it is clear that these embodiments may be practiced without these specific descriptions.

As used herein, the terms “component,” “module,” “system,” and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an implementation of software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. Additionally, these components can execute from various computer-readable media having various data structures stored thereon. Components can transmit signals, for example, with one or more data packets (e.g., data and/or signals from one component interacting with other components in a local system, a distributed system, to other systems and over a network such as the Internet). Depending on the data being transmitted, they may communicate through local and/or remote processes.

Additionally, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified or clear from context, “X utilizes A or B” is intended to mean one of the natural implicit substitutions. That is, either X uses A; X uses B; Or, if X uses both A and B, “X uses A or B” can apply to either of these cases. Additionally, the term “and/or” as used herein should be understood to refer to and include all possible combinations of one or more of the related listed items.

Additionally, the terms “comprise” and/or “comprising” should be understood to mean that the corresponding feature and/or element is present. However, the terms “comprise” and/or “comprising” should be understood as not excluding the presence or addition of one or more other features, elements and/or groups thereof. Additionally, unless otherwise specified or the context is clear to indicate a singular form, the singular terms herein and in the claims should generally be construed to mean “one or more.”

And, the term “at least one of A or B” should be interpreted to mean “a case containing only A,” “a case containing only B,” and “a case of combining A and B.”

Those skilled in the art will additionally recognize that the various illustrative logical blocks, components, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or a combination of both. It must be recognized that it can be implemented with To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software will depend on the specific application and design constraints imposed on the overall system. A skilled technician can implement the described functionality in a variety of ways for each specific application. However, such implementation decisions should not be construed as causing a departure from the scope of the present disclosure.

The description of the presented embodiments is provided to enable anyone skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Therefore, the present invention is not limited to the embodiments presented herein. The present invention is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

The description of the presented embodiments is provided to enable anyone skilled in the art to use or practice the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Therefore, the present invention is not limited to the embodiments presented herein. The present invention is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

In the present disclosure, terms represented by N, such as first, second, or third, are used to distinguish a plurality of entities. For example, the entities expressed as first and second may be the same or different from each other. Terms expressed as 1-1, 1-2, and 2-1, 2-2 may also be used to distinguish a plurality of entities.

In the present disclosure, visual localization refers to a technology for estimating the current location or pose of a device using images taken indoors or outdoors. Pose estimation of a device (or camera) refers to determining the translation and rotation information of a dynamically changing camera viewpoint.

The first image in the present disclosure includes, for example, an image captured by a photographing device, and the pose of the device corresponding to the first image is determined using the first image and a reference image stored in a server, etc. Or it can be estimated.

A reference image in the present disclosure may include a pre-stored image used for pose estimation of the device that captured the first image. The reference image may include actual images taken in a specific area to implement, for example, augmented reality or virtual reality. In one example, this reference image may be stored on computing device 100 or another computing device. Metadata mapped to the reference image can also be used to estimate the pose of the device.

Metadata may be used in performing visual localization in this disclosure. For example, metadata may include additional information other than the image, such as information related to the device, pixel coordinate information of feature points, descriptor of feature points, landmark information obtained from the image, character information obtained from the image, etc. . According to an embodiment of the present disclosure, by using various forms of metadata, comparison between images captured by a photographing device and pre-stored reference images can be made more smoothly, and further, in an accurate manner using fewer computing resources. Camera pose estimation can be performed.

Viewpoint transformation in the present disclosure may mean changing the location and/or viewing direction of a photographing device that photographs a specific object. For example, viewpoint transformation may mean transformation of the visual localization value of the shooting device. For example, viewpoint transformation converts the first image converted to be based on a three-dimensional camera coordinate system indicating the location of the photographing device and the direction in which the photographing device is looking from a first coordinate value on the camera coordinate system to the first coordinate value. This may mean converting to a different second coordinate value. For example, when performing viewpoint transformation, at least one of a rotation matrix and a translation vector may be used.

Object and subject in this disclosure may be used interchangeably.

1 schematically shows a block diagram of a computing device 100 according to an embodiment of the present disclosure.

Computing device 100 according to an embodiment of the present disclosure may include a processor 110 and a memory 130.

The configuration of the computing device 100 shown in FIG. 1 is only a simplified example. In one embodiment of the present disclosure, the computing device 100 may include different configurations for performing the computing environment of the computing device 100, and only some of the disclosed configurations may configure the computing device 100.

The computing device 100 in the present disclosure may refer to any type of node constituting a system for implementing embodiments of the present disclosure. Computing device 100 may refer to any type of user terminal or any type of server. The components of the computing device 100 described above are exemplary and some may be excluded or additional components may be included. For example, when the above-described computing device 100 includes a user terminal, an output unit (not shown) and an input unit (not shown) may be included within the scope of the computing device 100.

The client's device in the present disclosure may mean a device for acquiring an image. The first image may be included in information captured by the client device. The query image captured by the client device may be used, for example, for comparison with a reference image within the server, and thus the pose of the client device at the time of capture may be determined.

The computing device 100 in the present disclosure may perform technical features according to embodiments of the present disclosure, which will be described later. For example, the computing device 100 may generate and/or augment data for learning an artificial intelligence-based model.

Computing device 100 in the present disclosure obtains a first image and a first depth map from an imaging device, and applies a viewpoint transformation to the first image and the first depth map, thereby producing a transformed image and a transformed depth map. Obtain converted 3D data including, acquire first feature points corresponding to the converted image using a pre-trained artificial intelligence-based first feature point acquisition model, and obtain the converted depth map and the first feature point. 1 Obtaining restored 3D data by applying viewpoint inverse transformation to the converted image containing feature points, and generating labeling information corresponding to the first image based on the restored 3D data. can do. In the above-described manner, the computing device 100 can automatically generate accurate labeling information corresponding to the captured first image. In the above-described manner, a feature point extraction model that provides constant feature points and descriptors can be constructed even if the viewpoint for shooting changes.

In one embodiment, the processor 110 may consist of at least one core, including a central processing unit (CPU) and a general purpose graphics processing unit (GPGPU) of the computing device 100. , may include a processor for data analysis and/or processing, such as a tensor processing unit (TPU).

The processor 110 reads the computer program stored in the memory 130 and uses methods of visual localization according to an embodiment of the present disclosure (e.g., a methodology for building a feature point extraction model and/or a methodology for generating learning data for a feature point extraction model). ) can be performed. In addition, the processor 110 reads the computer program stored in the memory 130 and uses methods of visual localization according to an embodiment of the present disclosure (e.g., a methodology for building a feature point extraction model and/or learning data for a feature point extraction model). creation methodology) can be performed.

According to an embodiment of the present disclosure, the processor 110 may perform an operation for learning a neural network. The processor 110 is used for learning neural networks, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating the weights of the neural network using backpropagation. Calculations can be performed. At least one of the CPU, GPGPU, and TPU of the processor 110 may process learning of the network function. For example, CPU and GPGPU can work together to process learning of network functions and data classification using network functions. Additionally, in one embodiment of the present disclosure, processors of a plurality of computing devices may be used together to process learning of a network function and data classification using a network function. Additionally, a computer program executed in a computing device according to an embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program. Operations for a neural network according to an embodiment of the present disclosure will be described later with reference to FIG. 2.

Additionally, processor 110 may typically handle overall operations of computing device 100. For example, the processor 110 processes data, information, or signals input or output through components included in the computing device 100 or runs an application program stored in the storage to provide information or information appropriate to the user. Functions can be provided or processed.

According to one embodiment of the present disclosure, the memory 130 may store any type of information generated or determined by the processor 110 and any type of information received by the computing device 100. According to an embodiment of the present disclosure, the memory 130 may be a storage medium that stores computer software that allows the processor 110 to perform operations according to embodiments of the present disclosure. Accordingly, the memory 130 may refer to computer readable media for storing software code required to perform embodiments of the present disclosure, data to be executed by the code, and execution results of the code.

According to one embodiment of the present disclosure, the memory 130 may refer to any type of storage medium. For example, the memory 130 may be a flash memory type or a hard disk type. ), multimedia card micro type, card type memory (e.g. SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only) Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. The computing device 100 may operate in connection with web storage that performs a storage function of the memory 130 on the Internet. The description of the memory described above is only an example, and the memory 130 used in the present disclosure is not limited to the examples described above.

The communication unit (not shown) in the present disclosure can be configured regardless of the communication mode, such as wired or wireless, and can be used in various communication networks such as a personal area network (PAN) and a wide area network (WAN). It can be configured. In addition, the network unit 150 can operate based on the well-known World Wide Web (WWW), and is a wireless transmission technology used for short-distance communication such as Infrared Data Association (IrDA) or Bluetooth. You can also use .

Computing device 100 in the present disclosure may include any type of user terminal and/or any type of server. Accordingly, embodiments of the present disclosure may be performed by a server and/or a user terminal.

A user terminal may include any type of terminal capable of interacting with a server or other computing device. User terminals include, for example, mobile phones, smart phones, laptop computers, personal digital assistants (PDAs), slate PCs, tablet PCs, and ultrabooks. It can be included.

The photographing device in the present disclosure may include the user terminal described above. The imaging device may include modules for generating a depth map. Accordingly, the imaging device may generate an image and/or a depth map and transmit it to the computing device 100. In one embodiment, a photographing device may refer to any type of equipment for detecting an optical image, converting it into an electrical signal, and transmitting it to the computing device 100. For example, the imaging device may include at least one of a camera, scanner, Lidar, and/or vision sensor. In additional embodiments, the computing device 100 may include a photographing device or may be linked to an external device wirelessly or wired.

Servers may include any type of computing system or computing device, such as, for example, microprocessors, mainframe computers, digital processors, portable devices, and device controllers.

In an additional embodiment, the above-described server may include a storage unit (not shown) that stores and manages a reference image, metadata corresponding to the reference image, 3D map, etc. This storage may be included within the server or may exist under the management of the server. As another example, the storage unit may be implemented in a form that exists outside the server and can communicate with the server. In this case, the storage may be managed and controlled by an external server that is different from the server.

The computing device 100 according to some embodiments of the present disclosure may acquire feature points using various artificial intelligence-based models. For example, the computing device 100 may acquire a feature point corresponding to the first image using a feature point acquisition model, which is an artificial intelligence-based model.

In one embodiment, the feature point acquisition model may include a neural network built through deep learning or machine learning. The feature point acquisition model may acquire feature points from the image and/or depth map included in the input data.

For example, an image in the present disclosure may include at least one red-green-blue (RGB) image and/or at least one grayscale image.

For example, the depth map in the present disclosure may include an image or information indicating the relative distances of each pixel present in a specific image. Accordingly, the depth map may include information related to the distance from the location where the image is taken to the surface of the subject.

Figure 2 is a schematic diagram showing a network function according to an embodiment of the present disclosure.

In the present disclosure, a feature point acquisition model (eg, a first feature point acquisition model and/or a second feature point acquisition model) may correspond to an artificial intelligence-based model that will be described in detail below.

Throughout this disclosure, the terms artificial intelligence-based model, model, computational model, neural network, network function, and neural network may be used interchangeably. A neural network can generally consist of a set of interconnected computational units, which can be referred to as nodes. These nodes may also be referred to as neurons. A neural network consists of at least one node. Nodes (or neurons) that make up neural networks may be interconnected by one or more links.

Within a neural network, one or more nodes connected through a link may form a relative input node and output node relationship. The concepts of input node and output node are relative, and any node in an output node relationship with one node may be in an input node relationship with another node, and vice versa. As described above, input node to output node relationships can be created around links. One or more output nodes can be connected to one input node through a link, and vice versa.

In a relationship between an input node and an output node connected through one link, the value of the data of the output node may be determined based on the data input to the input node. Here, the link connecting the input node and the output node may have a weight. Weights may be variable and may be varied by the user or algorithm in order for the neural network to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. The output node value can be determined based on the weight.

As described above, in a neural network, one or more nodes are interconnected through one or more links to form an input node and output node relationship within the neural network. The characteristics of the neural network can be determined according to the number of nodes and links within the neural network, the correlation between the nodes and links, and the value of the weight assigned to each link. For example, if the same number of nodes and links exist and two neural networks with different weight values of the links exist, the two neural networks may be recognized as different from each other.

A neural network may consist of a set of one or more nodes. A subset of nodes that make up a neural network can form a layer. Some of the nodes constituting the neural network may form one layer based on the distances from the first input node. For example, a set of nodes with a distance n from the initial input node may constitute n layers. The distance from the initial input node can be defined by the minimum number of links that must be passed to reach the node from the initial input node. However, this definition of a layer is arbitrary for explanation purposes, and the order of a layer within a neural network may be defined in a different way than described above. For example, a layer of nodes may be defined by distance from the final output node.

The initial input node may refer to one or more nodes in the neural network through which data is directly input without going through links in relationships with other nodes. Alternatively, in a neural network network, in the relationship between nodes based on links, it may mean nodes that do not have other input nodes connected by links. Similarly, the final output node may refer to one or more nodes that do not have an output node in their relationship with other nodes among the nodes in the neural network. Additionally, hidden nodes may refer to nodes constituting a neural network other than the first input node and the last output node.

The neural network according to an embodiment of the present disclosure is a neural network in which the number of nodes in the input layer may be the same as the number of nodes in the output layer, and the number of nodes decreases and then increases again as it progresses from the input layer to the hidden layer. You can. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be less than the number of nodes in the output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. there is. In addition, the neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in the input layer may be greater than the number of nodes in the output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. You can. A neural network according to another embodiment of the present disclosure may be a neural network that is a combination of the above-described neural networks.

A deep neural network (DNN) may refer to a neural network that includes multiple hidden layers in addition to the input layer and output layer. Deep neural networks allow you to identify latent structures in data. In other words, it is possible to identify the potential structure of a photo, text, video, voice, or music (e.g., what object is in the photo, what the content and emotion of the text are, what the content and emotion of the voice are, etc.) . Deep neural networks include convolutional neural network (CNN), recurrent neural network (RNN), auto encoder, restricted Boltzmann machine (RBM), and deep trust network ( It may include deep belief network (DBN), Q network, U network, Siamese network, generative adversarial network (GAN), etc. The description of the deep neural network described above is only an example and the present disclosure is not limited thereto.

A neural network may be trained in at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Learning of a neural network may be a process of applying knowledge for the neural network to perform a specific operation to the neural network.

Neural networks can be trained to minimize output errors. In neural network learning, learning data is repeatedly input into the neural network, the output of the neural network and the error of the target for the learning data are calculated, and the error of the neural network is transferred from the output layer of the neural network to the input layer in the direction of reducing the error. This is the process of updating the weight of each node in the neural network through backpropagation. In the case of supervised learning, learning data in which the correct answer is labeled for each learning data is used (i.e., labeled learning data), while in the case of unsupervised learning, the correct answer may not be labeled in each learning data. That is, for example, in the case of supervised learning on data classification, the training data may be data in which each training data is labeled with a category. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the label of the training data. As another example, in the case of unsupervised learning on data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the reverse direction (i.e., from the output layer to the input layer) in the neural network, and the connection weight of each node in each layer of the neural network can be updated according to backpropagation. The amount of change in the connection weight of each updated node may be determined according to the learning rate. The neural network's calculation of input data and backpropagation of errors can constitute a learning cycle (epoch). The learning rate may be applied differently depending on the number of repetitions of the learning cycle of the neural network. For example, in the early stages of neural network training, a high learning rate can be used to increase efficiency by allowing the neural network to quickly achieve a certain level of performance, and in the later stages of training, a low learning rate can be used to increase accuracy.

In the learning of neural networks, the training data can generally be a subset of real data (i.e., the data to be processed using the learned neural network), and thus the error for the training data is reduced, but the error for the real data is reduced. There may be an incremental learning cycle. Overfitting is a phenomenon in which errors in actual data increase due to excessive learning on training data. For example, a phenomenon in which a neural network that learned a cat by showing a yellow cat fails to recognize that it is a cat when it sees a non-yellow cat may be a type of overfitting. Overfitting can cause errors in machine learning algorithms to increase. To prevent such overfitting, various optimization methods can be used. To prevent overfitting, methods such as increasing the learning data, regularization, dropout to disable some of the network nodes during the learning process, and use of a batch normalization layer can be applied. You can.

A computer-readable medium storing a data structure according to an embodiment of the present disclosure is disclosed.

Data structure can refer to the organization, management, and storage of data to enable efficient access and modification of data. Data structure can refer to the organization of data to solve a specific problem (e.g., retrieving data, storing data, or modifying data in the shortest possible time). A data structure may be defined as a physical or logical relationship between data elements designed to support a specific data processing function. Logical relationships between data elements may include connection relationships between user-defined data elements. Physical relationships between data elements may include actual relationships between data elements that are physically stored in a computer-readable storage medium (e.g., a persistent storage device). A data structure may specifically include a set of data, relationships between data, and functions or instructions applicable to the data. Effectively designed data structures allow computing devices to perform computations while minimizing the use of the computing device's resources. Specifically, computing devices can increase the efficiency of operations, reading, insertion, deletion, comparison, exchange, and search through effectively designed data structures.

Data structures can be divided into linear data structures and non-linear data structures depending on the type of data structure. A linear data structure may be a structure in which only one piece of data is connected to another piece of data. Linear data structures may include List, Stack, Queue, and Deque. A list can refer to a set of data that has an internal order. The list may include a linked list. A linked list may be a data structure in which data is connected in such a way that each data is connected in a single line with a pointer. In a linked list, a pointer may contain connection information to the next or previous data. Depending on its form, a linked list can be expressed as a singly linked list, a doubly linked list, or a circularly linked list. A stack may be a data listing structure that allows limited access to data. A stack can be a linear data structure in which data can be processed (for example, inserted or deleted) at only one end of the data structure. Data stored in the stack may have a data structure (LIFO-Last in First Out) where the later it enters, the sooner it comes out. A queue is a data listing structure that allows limited access to data. Unlike the stack, it can be a data structure (FIFO-First in First Out) where data stored later is released later. A deck can be a data structure that can process data at both ends of the data structure.

A non-linear data structure may be a structure in which multiple pieces of data are connected behind one piece of data. Nonlinear data structures may include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structure may include a tree data structure. A tree data structure may be a data structure in which there is only one path connecting two different vertices among a plurality of vertices included in the tree. In other words, it may be a data structure that does not form a loop in the graph data structure.

Throughout this specification, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used with the same meaning. Below, it is described in a unified manner as a neural network. Data structures may include neural networks. And the data structure including the neural network may be stored in a computer-readable medium. Data structures including neural networks also include data preprocessed for processing by a neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, neural network It may include a loss function for learning. A data structure containing a neural network may include any of the components disclosed above. In other words, the data structure including the neural network includes preprocessed data for processing by the neural network, data input to the neural network, weights of the neural network, hyperparameters of the neural network, data acquired from the neural network, activation functions associated with each node or layer of the neural network, neural network It may be configured to include all or any combination of the loss function for learning. In addition to the configurations described above, a data structure containing a neural network may include any other information that determines the characteristics of the neural network. Additionally, the data structure may include all types of data used or generated in the computational process of a neural network and is not limited to the above. Computer-readable media may include computer-readable recording media and/or computer-readable transmission media. A neural network can generally consist of a set of interconnected computational units, which can be referred to as nodes. These nodes may also be referred to as neurons. A neural network consists of at least one node.

The data structure may include data input to the neural network. A data structure containing data input to a neural network may be stored in a computer-readable medium. Data input to the neural network may include learning data input during the neural network learning process and/or input data input to the neural network on which training has been completed. Data input to the neural network may include data that has undergone pre-processing and/or data subject to pre-processing. Preprocessing may include a data processing process to input data into a neural network. Therefore, the data structure may include data subject to preprocessing and data generated by preprocessing. The above-described data structure is only an example and the present disclosure is not limited thereto.

The data structure may include the weights of the neural network. In this disclosure, weights and parameters may be used interchangeably. The data structure including the weights of the neural network may be stored in a computer-readable medium. A neural network may include multiple weights. Weights may be variable and may be varied by the user or algorithm in order for the neural network to perform the desired function. For example, when one or more input nodes are connected to one output node by respective links, the output node is set to the values input to the input nodes connected to the output node and the links corresponding to each input node. Based on the weight, the data value output from the output node can be determined. The above-described data structure is only an example and the present disclosure is not limited thereto.

As an example and not a limitation, the weights may include weights that are changed during the neural network learning process and/or weights for which neural network learning has been completed. Weights that change during the neural network learning process may include weights that change at the start of the learning cycle and/or weights that change during the learning cycle. Weights for which neural network training has been completed may include weights for which a learning cycle has been completed. Therefore, the data structure including the weights of the neural network may include weights that are changed during the neural network learning process and/or the data structure including the weights for which neural network learning has been completed. Therefore, the above-mentioned weights and/or combinations of each weight are included in the data structure including the weights of the neural network. The above-described data structure is only an example and the present disclosure is not limited thereto.

The data structure including the weights of the neural network may be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be the process of converting a data structure into a form that can be stored on the same or a different computing device and later reorganized and used. Computing devices can transmit and receive data over a network by serializing data structures. Data structures containing the weights of a serialized neural network can be reconstructed on the same computing device or on a different computing device through deserialization. The data structure including the weights of the neural network is not limited to serialization. Furthermore, the data structure including the weights of the neural network is a data structure to increase computational efficiency while minimizing the use of computing device resources (e.g., in non-linear data structures, B-Tree, Trie, m-way search tree, AVL tree, Red-Black Tree) may be included. The foregoing is merely an example and the present disclosure is not limited thereto.

The data structure may include hyper-parameters of a neural network. And the data structure including the hyperparameters of the neural network can be stored in a computer-readable medium. A hyperparameter may be a variable that can be changed by the user. Hyperparameters include, for example, learning rate, cost function, number of learning cycle repetitions, weight initialization (e.g., setting the range of weight values subject to weight initialization), Hidden Unit. It may include a number (e.g., number of hidden layers, number of nodes in hidden layers). The above-described data structure is only an example and the present disclosure is not limited thereto.

Figure 3 exemplarily shows a method of generating labeling information for an image based on viewpoint transformation according to an embodiment of the present disclosure.

The flowchart shown in FIG. 3 may be performed, for example, by computing device 100.

In one embodiment, computing device 100 may obtain a first image and a first depth map from a photography device (310).

For example, computing device 100 may provide an augmented reality and/or virtual reality platform. The computing device 100 may estimate the current location and direction of the photographing device by obtaining photographed data from the photographing device. Computing device 100 may allow a client to enjoy activities on an augmented reality and/or virtual reality platform by estimating the location and orientation of the imaging device.

In one embodiment, the first image acquired by the photographing device may include a two-dimensional RGB image or a grayscale image.

In one embodiment, the first depth map acquired by the photographing device may include an image or information indicating the relative distances of each pixel present in a specific image. Accordingly, this depth map may include information related to the distance from the location of the imaging device that captures the image to the surface of the subject.

In a further embodiment, computing device 100 may obtain additional information of the photographing device (eg, camera information of the photographing device). For example, additional information about the imaging device may be acquired together when acquiring the first image and the first depth map. Additional information may include location information of the photographing device. Location information is, for example, geodetic datum or geodetic system (or geodetic reference datum, geodetic reference system, or geodetic reference frame), spatial reference system (SRS) or coordinate reference system (CRS) used in a specific region or country. , or may include coordinate values and/or direction values for a geographic coordinate system (GCS), etc. Other examples of location information may include GPS.

Additionally, the additional information about the photographing device may further include hardware-related information of the photographing device, such as focal length, principal point, resolution, etc. In addition, if there is a record of estimating the past camera pose of the device, the additional information of the device includes preliminary estimation information about the camera pose of the device calculated from the value of the IMU (Inertial Measurement Unit) of the device. It can be included. The preliminary estimation information about the camera pose of the photographing device may include an approximate camera pose estimated using IMU information, and this preliminary estimation information may include approximate position information and direction information about the device. When such preliminary estimation information is used, the candidate group of reference images to be matched with the captured image can be reduced, so comparison between the captured image and the reference image stored in the server (or computing device) can be efficiently performed. Accordingly, resource-efficient visual localization can be achieved. The methodology for visual localization is specifically described in Korean Patent Application Nos. 10-2022-0081908 and 10-2022-0078715, which are incorporated by reference in their entirety into this patent application.

In one embodiment of the present disclosure, the computing device 100 uses additional information of the photographing device to determine candidate reference images that are subject to feature point matching among a plurality of reference images in a database, and the determined candidate reference images. Visual localization, including pose estimation for the camera of the photographing device, can be performed based on feature point matching between the captured images and the photographed image.

In one embodiment of the present disclosure, the additional information of the device may further include camera information of the photographing device, and an example of such camera information may include intrinsic parameters of the camera.

In one embodiment, computing device 100 may perform viewpoint transformation using internal parameters. This viewpoint transformation may include transformation in a three-dimensional coordinate system.

In an additional embodiment, the computing device 100 may determine a pose estimation algorithm used to perform the visual localization among a plurality of pose estimation algorithms based on the camera information of the photographing device. For example, pose estimation for a device (eg, camera) may estimate 6DoF in a direction that minimizes reprojection error.

For example, when the camera's internal parameters are not given (i.e., an uncalibrated camera), the pose estimation algorithm uses the Direct Linear Transformation (DLT) methodology to determine the elements of the rotation matrix and the translation vector. The reprojection error equation, which includes the elements of (vector) elements and the internal parameters of the camera as unknowns, can be changed to a linear equation and the rotation matrix, translation vector, and camera matrix can be estimated through QR decomposition.

For example, given the camera's internal parameters (i.e., a calibrated camera), the pose estimation algorithm uses the Perspective-n-Point (PnP) methodology to transform the reprojection error, a non-linear equation, into a linear equation, which is different from DLT. Similarly, pose estimation can be performed by directly solving the reprojection error, which is a non-linear equation, using the Gauss-Newton methodology or the Levenberg-Marquardt methodology. In this example, representative PnP methodologies may include P3P, EPnP, and SQPnP methodologies.

In a further example, the DLT and PnP methodologies described above may introduce RANSAC to minimize camera pose errors resulting from inaccuracies in the 3D coordinates of reference feature points or inaccuracies in feature points.

Therefore, as described above, the technique according to an embodiment of the present disclosure can apply the camera pose estimation algorithm differently depending on the presence or absence of camera information, for example, DLT or PnP depending on the presence or absence of camera information. By using the methodology, camera pose estimation can be performed.

In one embodiment, the computing device 100 may obtain transformed 3D data including a transformed image and a transformed depth map by applying viewpoint transformation to the first image and the first depth map (320) ).

Viewpoint conversion in the present disclosure may mean changing the viewpoint of the imaging device. In one embodiment, viewpoint transformation may include changing the location and viewing direction of the imaging device. Viewpoint transformation in the present disclosure may mean changing a three-dimensional value in the three-dimensional camera coordinate system of the photographing device. In one embodiment, the viewpoint transformation converts the converted first image to be based on a three-dimensional camera coordinate system indicating the location of the photographing device and the direction in which the photographing device is looking from a first coordinate value on the camera coordinate system that is different from the first coordinate value. This may mean converting to a second coordinate value. This viewpoint transformation can be performed using a rotation matrix and translation vector. If at least one of the rotation matrix and translation vector is different, different viewpoint transformations may be achieved.

For example, computing device 100 may apply a plurality of viewpoint transformations to the first image and the first depth map where at least one of the rotation matrix and the translation vector is different, thereby creating a plurality of transformed images and a plurality of transformed depths. It may include obtaining a plurality of converted 3D data including maps. In this example, the computing device 100 may acquire a plurality of viewpoint converted data corresponding to the captured first image.

For example, in viewpoint transformation, a unique transformation matrix may be applied to each pixel of the first image. In this example, viewpoint transformation may be performed for each pixel on the first image through viewpoint transformation.

In one embodiment, viewpoint transformation may mean transformation between two coordinate values in a camera coordinate system. In one embodiment, viewpoint transformation may correspond to a warp operation on two-dimensional data and/or three-dimensional data.

According to an embodiment of the present disclosure, the computing device 100 may obtain various captured data whose viewpoints have been transformed by applying a plurality of viewpoint transformations to the captured data acquired by the photographing device.

In one embodiment, the converted 3D data may include an image to which viewpoint transformation has been applied and a depth map to which viewpoint transformation has been applied.

In one embodiment, computing device 100 may convert a first image having a first coordinate system into a second coordinate system and convert a first image having a second coordinate system into a third coordinate system. Viewpoint transformation may be applied to the first image having the third coordinate system, and the viewpoint transformed first image may be acquired based on the third coordinate system. For example, the first coordinate system, the second coordinate system, and the third coordinate system may mean coordinate systems in which factors constituting the coordinate system are different from each other. For example, at least one of the first coordinate system and the second coordinate system may correspond to a two-dimensional coordinate system and the third coordinate system may correspond to a three-dimensional coordinate system.

In the above-described manner, the computing device 100 can perform three-dimensional viewpoint transformation through transformation from the first coordinate system to the second coordinate system and from the second coordinate system to the third coordinate system.

In one embodiment, the computing device 100 may convert the first image having a pixel coordinate system into a normalized coordinate system using camera information. The computing device 100 may convert the first image having the normalized coordinate system into a three-dimensional camera coordinate system using the acquired depth map. The computing device 100 may obtain the transformed 3D data including the transformed image and the transformed depth map by applying viewpoint transformation to the first image having the camera coordinate system.

In one embodiment, the pixel coordinate system may refer to a coordinate system indicating the location of a pixel on an image obtained when a photographing device captures the image. The camera coordinate system may refer to a coordinate system determined by the location of the photographing device and the direction in which it is looking. For example, in the camera coordinate system, the direction the camera faces can be set to the Z-axis. The normalized coordinate system may refer to coordinates obtained when points are projected onto a plane with a Z value of 1 in the camera coordinate system. For example, the origin in the normalized coordinate system may correspond to (0,0,1) in the camera coordinate system. Projection in a normalized coordinate system may mean projection along a line segment connecting the center point of the photographing device and the position of a point in a plane space where the Z value of the camera coordinate system is 1.

In one embodiment, conversion from a normalized coordinate system to a pixel coordinate system may be performed using camera information (eg, internal parameters of the camera including focal length, principal point, skew parameter, etc.). For example, the value in the pixel coordinate system can be obtained by multiplying the camera information in the normalized coordinate system by the corresponding camera matrix. In this example, the focal length determines the difference in scale between the normalized coordinate system and the camera coordinate system, the principal point moves the origin of the pixel coordinate system to the upper left on the image plane, and the skew parameter determines the horizontal and vertical dimensions of the image. It shows asymmetry.

In one embodiment, the computing device 100 may acquire first feature points corresponding to the converted image using a pre-trained artificial intelligence-based first feature point acquisition model (330).

In one embodiment, the first feature point acquisition model may correspond to a pre-trained model based on supervised learning to recognize as a feature point a point where the amount of change in at least one of color or geometric pattern in the subject exceeds a predetermined threshold. You can.

In one embodiment, the computing device 100 uses a pre-trained first feature point acquisition model to generate at least one feature point and/or at least one descriptor corresponding to each of the at least one feature point from the first image. It can be obtained. For example, a feature point may be the coordinates for each characteristic part or point in an image. For example, the descriptor may include at least one of information about the directionality and size of each feature point, and/or the relationship between pixels surrounding each feature point.

In one embodiment, the feature point acquisition model corresponds to a pre-trained model based on supervised learning to recognize as a feature point a point where the amount of change in at least one of color or geometric pattern in an object in the image exceeds a predetermined threshold. It can be. For example, the feature point acquisition model may correspond to a pre-trained model using an image-based neural network. As another example, the feature point acquisition model may correspond to a pre-trained model using a Transformer-based neural network.

As described above, according to an embodiment of the present disclosure, the computing device 100 may acquire feature points (and descriptors) for an image on which viewpoint conversion has been performed.

In one embodiment, the computing device 100 may obtain two-dimensional data by converting the transformed three-dimensional data having a three-dimensional camera coordinate system into a normalized coordinate system using a depth map transformed based on viewpoint transformation. there is. The computing device 100 may obtain the converted image by converting the two-dimensional data having the normalized coordinate system into a pixel coordinate system based on camera information of the photographing device obtained from the photographing device. The computing device 100 may input the converted image into the first feature point acquisition model to obtain first feature points corresponding to the converted image.

As described above, the first feature point acquisition model may be operated by inputting an image having a coordinate system different from the coordinate system used to perform viewpoint transformation. For example, the coordinate system used as an input by the first feature point acquisition model may correspond to a two-dimensional coordinate system, and the coordinate system to which viewpoint transformation is applied may correspond to a three-dimensional coordinate system. For example, the first feature point acquisition model can operate in a pixel coordinate system and the viewpoint transformation can be applied in a 3D camera coordinate system.

Accordingly, the computing device 100 performs transformation from the third coordinate system in which the viewpoint transformation was performed to the second coordinate system, and converts the result of the transformation from the second coordinate system to the first coordinate system into the first feature point acquisition model. It can be used as input data. The first feature point acquisition model may output feature points (and/or descriptors) corresponding to input data having this first coordinate system. The output of the first feature point acquisition model may include feature points and descriptors expressed in the image having the first coordinate system. Here, the first coordinate system, the second coordinate system, and the third coordinate system may mean coordinate systems in which the components constituting the coordinate system are different. For example, at least one of the first and second coordinate systems may include a two-dimensional coordinate system, and the third coordinate system may include a three-dimensional coordinate system.

In one embodiment, the computing device 100 may obtain restored 3D data by applying viewpoint inverse transformation to the converted image including the converted depth map and the first feature points (340).

In one embodiment, viewpoint inverse-transformation may refer to a transformation corresponding to the inverse of a viewpoint transformation. For example, when viewpoint transformation and viewpoint inverse transformation are performed on a specific image with a specific viewpoint, the specific image with a specific viewpoint can be restored.

In one embodiment, the viewpoint inverse transformation comprises a rotation matrix (R-1) corresponding to the inverse of the rotation matrix (R) used in the viewpoint transformation, and a translation vector (T You can use the translational vector (-R-1 * T) corresponding to the inverse of ).

In one embodiment, the viewpoint inverse transformation may be applied to a coordinate system that is different from the coordinate system in which the first feature point acquisition model is operable. For example, the viewpoint inverse transformation may be applied in a three-dimensional camera coordinate system, and the first feature acquisition model may be operable in a two-dimensional pixel coordinate system.

In one embodiment, the viewpoint inversion can be applied in the same coordinate system as the viewpoint transformation is applicable. The process of viewpoint inverse transformation can be applied to the output of the first feature point acquisition model. As described above with respect to viewpoint transformation, computing device 100 performs transformation from the first coordinate system corresponding to the output of the first feature point acquisition model to the second coordinate system, and from the second coordinate system to the third coordinate system. After performing the transformation, viewpoint inverse transformation can be applied to the data. As described above, the first coordinate system, the second coordinate system, and the third coordinate system may mean coordinate systems in which elements constituting the coordinate system are different from each other. For example, at least one of the first and second coordinate systems may include a two-dimensional coordinate system, and the third coordinate system may include a three-dimensional coordinate system.

In one embodiment, the computing device 100 may obtain the converted image including the first feature points by combining the first feature points with the converted image. The computing device 100 may convert the converted image including the first feature points into a normalized coordinate system using camera information of the photographing device obtained from the photographing device. Using the converted depth map, the converted image including the first feature points having the normalized coordinate system can be converted into a three-dimensional camera coordinate system. The computing device 100 may obtain the restored 3D data by applying viewpoint inverse transformation to the converted image including the first feature points having the 3D camera coordinate system.

In one embodiment, the computing device 100 may generate labeling information corresponding to the first image based on the restored 3D data (350).

In one embodiment, the first image captured by the imaging device may be acquired in an unlabeled state. The computing device 100 may generate labeling information indicating feature points corresponding to the first image using the restored 3D data.

In one embodiment, the computing device 100 applies a reconstructed depth map in the reconstructed 3D data to the reconstructed 3D data, thereby normalizing the reconstructed 3D data having a 3D camera coordinate system to 2D. Two-dimensional restored data with a coordinate system can be obtained. The computing device 100 converts the two-dimensional restored data into a pixel coordinate system based on applying camera information of the photographing device obtained from the photographing device to the two-dimensional reconstructed data having a two-dimensional normalized coordinate system. , a restored image can be obtained. The computing device 100 may generate labeling information corresponding to the first image based on the reconstructed image having the pixel coordinate system.

As described above, a coordinate system change may be made for the result of the viewpoint inverse transformation corresponding to a three-dimensional coordinate system. For example, a transformation can be made from a third coordinate system to a second coordinate system and a transformation from the second coordinate system to a first coordinate system. For example, the first coordinate system may correspond to a coordinate system that can be used as an input to the first feature point acquisition model. The result converted to the first coordinate system may represent a first image including feature points (and/or descriptors). Depending on the implementation aspect, results converted to the first coordinate system may include a plurality of images. Feature points and descriptors corresponding to each of the plurality of images may be included. The optimal feature points for the first image can be determined as labeling information (i.e., correct answer information) through a combination of feature points included in a plurality of images.

In one embodiment, viewpoint transformation may be applied to the captured first image and the first depth map to generate a plurality of transformed data. The computing device 100 may input the converted image included in the plurality of converted data into a pre-trained first feature point extraction model to obtain a plurality of data in which the first feature points are reflected. The computing device 100 may perform restoration of the first images in which the first feature points are reflected by applying viewpoint inverse transformation to a plurality of data in which the first feature points are reflected, and the computing device 100 may perform restoration of the first images in which the first feature points are reflected. By combining the first feature points included in the resulting first images, the feature points for the first image can be generated as labeling information. In this way, more accurate labeling information for the first image can be generated. Additionally, in this way, labeling information for the first image can be automatically generated.

In one embodiment, the computing device 100 applies a first coordinate system transformation to the photographed data, applies a viewpoint transformation to the result of the coordinate system transformation, applies a second coordinate system transformation to the result of the viewpoint transformation, and applies the coordinate system transformation to the result of the coordinate system transformation. Obtain feature points corresponding to the results, apply the third coordinate system transformation to the photographed data reflecting the feature points, apply viewpoint inverse transformation to the coordinate system transformed result, and apply fourth coordinate system transformation to the result of viewpoint inverse transformation. And, based on the results of applying the coordinate system transformation, learning data including labeling information for the photographed data can be automatically generated. In this example, the coordinate system transformation directions of the first coordinate system transformation and the third coordinate system transformation may correspond to each other, and the coordinate system transformation directions of the second coordinate system transformation and the fourth coordinate system transformation may correspond to each other.

In one embodiment, the computing device 100 uses a learning dataset including a first image and labeling information for the first image to learn a second feature point extraction model that extracts constant feature points even if the viewpoint changes. You can. The learning process for the second feature point extraction model will be described later in FIG. 7.

Since the labeling information generation method according to an embodiment of the present disclosure can obtain more accurate feature points for the captured image as labeling information, more robust learning of the feature point extraction model used in performing visual localization can be achieved. there is. Furthermore, more accurate inference may be possible based on more robust learning of the feature point extraction model used to perform visual localization.

Figure 4 exemplarily shows a method of generating labeling information for an image based on viewpoint transformation according to an embodiment of the present disclosure.

The methods shown in FIG. 4 may be performed, for example, by computing device 100.

In one embodiment, viewpoint transformation 430 may be applied to the first image 410 and the first depth map 420 corresponding to the subject acquired from the photographing device. For example, the computing device 100 may acquire the first image 410 and the first depth map 420 obtained from the photographing device. As another example, the computing device 100 may function as a photographing device.

In one embodiment, the viewpoint transformation 430 may include changing the first image 410 and the first depth map 420 acquired according to the first viewpoint to a second viewpoint on the three-dimensional camera coordinate system. You can. According to the viewpoint conversion 430, the converted first image 440a and the converted first depth map 440b may be obtained. For example, a viewpoint transformation 430 is applied to the first image 410 to produce a transformed first image 440a, and a viewpoint transformation 430 is applied to the first depth map 420. is applied to generate the converted first depth map 440b.

In one embodiment, the converted first image 440a may be input to the first feature point acquisition model 450. The first feature point acquisition model 450 is a pre-trained model based on artificial intelligence and can determine feature points on the input image. The first feature point acquisition model 450 may recognize a point where the amount of change in at least one of the color or geometric pattern of the subject exceeds a predetermined threshold as a feature point. The first feature point acquisition model 450 can recognize a plurality of feature points that can determine the appearance of a subject included in an input image. The first feature point acquisition model 450 may receive the converted first image 440a as input and output first feature points 460. The first feature points 460 may include feature points in the first image 440a that has undergone viewpoint conversion of the captured first image 410.

In one embodiment, the first feature point acquisition model 450 may include a machine learning-based neural network capable of extracting feature points from the image. For example, the first feature point acquisition model 450 may correspond to a model pre-trained using a supervised learning method based on a learning data set that uses the vertices of geometric patterns as ground truth for feature points.

In one embodiment, computing device 100 may apply viewpoint inverse transformation 470 to the transformed first image 440a including first feature points 460 and the transformed first depth map 440b. there is. The first feature points 460 extracted by the first feature point acquisition model 450 may be reflected on the converted first image 440a. Viewpoint inverse transformation 470 may be applied to the transformed first image 440a and the transformed first depth map 440b including the first feature points 460, and this viewpoint inverse transformation 470 is applied to the viewpoint It may mean the inverse of transformation 430. As a result of the viewpoint inverse transformation 470, restored 3D data 480 may be generated. For example, the restored 3D data 480 may correspond to the first image 410 including first feature points. As an example, the restored 3D data 480 may include a first depth map 420. For example, the restored 3D data 480 may include a first image 410 including first feature points and a first depth map 420.

In one embodiment, the computing device 100 may generate labeling information 490 for the first image 410 based on feature points included in the restored 3D data 480. Labeling information 490 may include information about feature points corresponding to the first image 410. The labeling information 490 may include information about feature points and/or information about descriptors obtained by performing viewpoint transformation 430 and inverse viewpoint transformation 470 of the first image 410. This labeling information 490 may be included in the learning data set used in the learning process of the second feature point acquisition model.

According to one embodiment of the present disclosure, information 420, 440b related to the depth map may be used in viewpoint transformation 430 and viewpoint inverse transformation 440b. As information 420 and 440b related to the depth map is used, viewpoint transformation 430 and viewpoint inverse transformation 440b on 3D camera coordinates can be performed efficiently.

In one embodiment, the viewpoint transformation 430 includes a plurality of transformed first images 440a and a plurality of transformed first depth maps 440b for the first image 410 and the first depth map 420. can be created. In this case, multiple different viewpoint transforms 430 may be used, and multiple different viewpoint inverse transforms 470 may be used. Accordingly, the restored 3D data 480 may include a plurality of sets of feature points corresponding to the first image 410, and based on the plurality of sets of feature points, the first image 410 Labeling information 490 may be generated. Labeling information 490 corresponding to the first image 410 may be generated based on a plurality of restored 3D data 480 corresponding to each of the plurality of converted 3D data 440a and 440b. For example, all feature points of a plurality of sets corresponding to the first image 410 may be used as labeling information 490. As another example, some of the feature points of the plurality of sets corresponding to the first image 410 may be used as labeling information 490. In this example, among the feature points of the plurality of sets corresponding to the first image 410, overlapping feature points may be used as labeling information 490.

As described above, the technique according to an embodiment of the present disclosure is a feature point obtained by applying a plurality of viewpoint transformations 430 and a plurality of viewpoint inverse transformations 480 to the photographed data 410 and 420. Since labeling information 490 can be generated based on For example, the accuracy of feature point information and descriptor information) can also be improved. Accordingly, a second feature point acquisition model that can output feature points and descriptors with high consistency even if the viewpoint changes can be created.

Figure 5 exemplarily illustrates viewpoint conversion according to an embodiment of the present disclosure.

In one embodiment, an image and depth map for the subject 510 may be acquired by an imaging device having a first viewpoint 530 on three-dimensional camera coordinates. In one embodiment, camera information about the photographing device having the first viewpoint 530 on 3D camera coordinates may be additionally obtained. The first viewpoint 530 may correspond to the first camera coordinate system 550.

In one embodiment, according to viewpoint transformation, the image and depth map for the subject 510 with the first viewpoint 530 are converted into the image and depth map for the subject 510 with the second viewpoint 520. can be changed. The changed image and depth map may correspond to the second camera coordinate system 540. The second viewpoint 520 may correspond to the second camera coordinate system 540.

In one embodiment, viewpoint conversion may be performed by Equation 1 below.

In one embodiment, may mean a camera coordinate value at the first viewpoint 530 corresponding to the first camera coordinate system 550. In one embodiment, in Equation 1 represents the camera coordinate value at the second viewpoint 520 corresponding to the second camera coordinate system 540.

In one embodiment, R in Equation 1 may represent a rotation matrix and T may represent a translation vector. For example, the viewpoint transformation may be different depending on the values of R and T. For example, R and T may refer to factors for expressing relative camera poses for viewpoint conversion.

Accordingly, at the first viewpoint 530 corresponding to the first camera coordinate system 550 Viewpoint transformation can be achieved by multiplying by the rotation matrix R and summing the translation vector T. The result of this viewpoint transformation This can be obtained.

Equation 2 exemplarily represents the matrix operation of viewpoint transformation according to Equation 1. Each of X ₁ , Y _{1 ,} and Z ₁ in the first matrix, which is the result of the operation in Equation 2, may represent the Each of X ₂ , Y _{2 ,} and Z ₂ in the second matrix that is the target of calculation may represent X-axis, Y-axis, and Z-axis values in the first camera coordinate system 550. The elements of r ₁₁ , r ₁₂ , r ₁₃ , r ₂₁ , r ₂₂ , r ₂₃ , r ₃₁ , r ₃₂ and r ₃₃ included in the third matrix representing the viewpoint transformation in Equation 2 are the rotation of the viewpoint transformation. It means matrix. Among the rotation matrices, r _{11 , r 2} 1 and r ₃₁ are calculated with X ₂ , which _is the value of the It can be calculated with Y ₂ , which is the value of the Y-axis, and r ₁₃ , r ₂₃ , and r ₃₃ among the rotation matrices can be calculated with Z ₂ , which is the value of the Z-axis in the first camera coordinate system 550 . The magnitude of the value of r _nk in the third matrix and the quantitative value of the rotation in the viewpoint transformation may be correlated. The larger the value of r _nk , the higher the intensity of rotational motion. Here, n and k are natural numbers with values from 1 to 3.

In one embodiment, t ₁ , t ₂ , and t ₃ in the third matrix of Equation 2 may mean elements representing translational vectors. The magnitudes of t ₁ , t ₂ and t ₃ and the quantitative value of translation in viewpoint translation can be correlated. The larger the value of t _n , the higher the intensity of translational motion. In one embodiment, element 1 in the first matrix, element 1 in the second matrix, and elements (0, 0, 0, 1) in the third matrix are used to express the sum of translational vectors in the form of a product. These are the elements used. Here, n is a natural number with values from 1 to 3.

Alternatively, the first matrix _of X ₁ , Y ₁ and Z ₁ is the product _of the third matrix of r _nk and _the second _{matrix of} It can also be expressed as a sum of translation matrices.

As shown in FIG. 5, viewpoint transformation may be performed on a three-dimensional camera coordinate system, and inverse transformation of the viewpoint may mean the inverse of the transformation shown in FIG. 5.

As shown in Figure 5, viewpoint transformation between two 3D camera coordinate systems can be implemented using a rotation matrix containing reflection that changes the direction of the axis and a translation vector containing a 3D vector that generates translational motion. there is. This viewpoint transformation may operate based on, for example, Euclidean transformation, Rigid transformation, and/or Isometry transformation. As shown in FIG. 5 , the viewpoint can be converted from the first image based on the first camera coordinate system 550 to the first image based on the second camera coordinate system 540.

FIG. 6 exemplarily illustrates transformation of coordinate systems based on viewpoint transformation according to an embodiment of the present disclosure.

In performing viewpoint transformation and/or viewpoint inverse transformation, transformation between coordinate systems may be involved. Figure 6 exemplarily shows processes in which coordinate systems are changed when performing viewpoint transformation.

In one embodiment of the present disclosure, there may be three different coordinate systems (eg, a first coordinate system, a second coordinate system, and a third coordinate system). The computing device 100 performs a coordinate system transformation in a first direction having directions from a first coordinate system to a second coordinate system and a third coordinate system, and a coordinate system transformation in a second direction having directions from a third coordinate system to a second coordinate system and the first coordinate system. It can be done. For example, coordinate system transformation in the first direction and coordinate system transformation in the second direction may be performed alternately. In this example, the pattern of performing coordinate system transformation in the first direction and coordinate system transformation in the second direction may be repeated twice. In the process of performing coordinate system transformations, feature points (and/or descriptors) for an image captured by a photographing device may be obtained. For example, in the process of performing coordinate system transformations, a plurality of viewpoint transformations may be applied to the image captured by the photographing device, and feature points (and/or descriptors) corresponding to the images according to the plurality of viewpoint transformations sets) can be obtained. Through the combination of these plural sets of feature points, optimal feature points for an image captured by a photographing device can be determined. In this way, the computing device 100 can automatically and accurately generate labeling information for images captured by the photographing device.

As shown in FIG. 6, the first pixel coordinate system 610 may represent the coordinate system of the first image captured by the photographing device. The computing device 100 may convert the first pixel coordinate system 610 into the first normalized coordinate system 620 using camera information of the photographing device. For example, a photographing device capable of supporting virtual reality or augmented reality has camera information including camera internal parameters, and the computing device 100 uses the camera information of the photographing device to create a first pixel coordinate system 610. The first image of can be converted to the first image of the first normalized coordinate system 620.

In one embodiment, the depth map 625 acquired together with the first image may be used to convert the first image of the first normalized coordinate system 620 into the first image of the first camera coordinate system 630. The depth map 625 may represent the Z value of a feature point. Information according to the depth map 625 can be used to express the first image in three dimensions. Information according to the depth map 625 may be used to express feature points in three dimensions. The computing device 100 may convert the first image of the first normalized coordinate system 620 into the first image of the first camera coordinate system 630 using the depth map 625.

In one embodiment, a viewpoint transformation 635 may be applied to the first image in the first camera coordinate system 630. The first image of the second camera coordinate system 640 may be obtained according to the result of applying the viewpoint transformation 635. The second camera coordinate system 640 has the same coordinate system as the first camera coordinate system 630, but coordinate values within the coordinate system may be different from those of the first camera coordinate system 630. For example, an area within the image (and/or depth map) to which rotation and/or translation can be applied to minimize image loss during the viewpoint transformation 635 may be predetermined. Within that region, rotation matrices and translation vectors can be determined, for example based on random sampling.

In one embodiment, a projection may be applied to the first image in the second camera coordinate system 640. Projection means transforming the first image in the second camera coordinate system 640 into a first image in the normalized coordinate system 650 using a transformed depth map (i.e., a viewpoint transformed depth map). Projection may mean dividing the first image in the second camera coordinate system 640 by the converted depth value. For example, projection may include dividing the location of each point of the first image in the second camera coordinate system 640 into each transformed depth map. As projection is applied to the first image of the second camera coordinate system 640, the first image of the second normalized coordinate system 650 may be obtained.

The computing device 100 may acquire the first image of the second pixel coordinate system 660 from the first image of the second normalized coordinate system 650 using the camera information 655 of the photographing device. The first image of the second pixel coordinate system 660 corresponds to a two-dimensional image to which three-dimensional viewpoint transformation has been applied. The first image of the second pixel coordinate system 660 is an image of the same dimension (eg, two-dimensional) as the first image of the first pixel coordinate system 660. The first image of the second pixel coordinate system 660 may have coordinate values that are different from the coordinate values of the first image of the first pixel coordinate system 660. In the present disclosure, the expression different coordinate systems may mean that the factors constituting the coordinate system (eg, the number of axes and/or the direction of the axes, etc.) are different. In the present disclosure, the expression different coordinate values may mean that the position values of coordinates expressed within the same coordinate system are different.

In one embodiment, the computing device 100 may perform feature point extraction for the first image in the second pixel coordinate system 660. This feature point extraction may be performed using a first feature point extraction model. The result of conversion from the first image in the pixel coordinate system 610 to the first image in the second pixel coordinate system 660 may be used as an input to the first feature point extraction model.

In one embodiment, the first image of the second camera coordinate system 640 may be acquired in the opposite direction from the first image of the second pixel coordinate system 660 including the feature points, and the first image of the second camera coordinate system 640 may be obtained. 1 Viewpoint inverse transformation may be performed on the image. After the viewpoint inverse transformation is performed, a first image of the first camera coordinate system 640 including feature points may be obtained. Then, conversion to the first normal coordinate system 620 and the first pixel coordinate system 610 is performed, and feature points for the first image of the first pixel coordinate system 610 can be obtained. Multiple sets of feature points may be obtained for a plurality of images in the first pixel coordinate system 610 (i.e., multiple images that have been viewpoint-transformed and viewpoint-de-transformed from the first image). Based on these feature points, labeling information corresponding to the first image can be obtained.

Figure 7 exemplarily shows a learning method of a second feature point acquisition model according to an embodiment of the present disclosure.

In one embodiment, viewpoint transformation 430 may be performed on the first depth map 420 and the first image 410, which are acquired by the imaging device. For specific features of the viewpoint conversion 430, refer to the details described in FIGS. 3 to 6 for convenience of explanation.

In one embodiment, as a result of the viewpoint transformation 430, a transformed first image 440a and a transformed first depth map 440b may be obtained. The converted first image 440a can be used as an input to the second feature point acquisition model 710. For example, the second feature point acquisition model 710 may refer to an artificial intelligence-based feature point extraction model that has a neural network structure corresponding to the first feature point acquisition model described above. The operation of the second feature point acquisition model 710 may include viewpoint inverse transformation. Accordingly, feature points for the transformed first image 440a are obtained and view-point inverse transformation is applied to the first image including the feature points, thereby generating second feature points 720b.

In one embodiment, the first image 410 may be used as an input to the second feature point acquisition model 710.

In one embodiment, the second feature point acquisition model 710 generates second feature points 720b in response to the converted first image 440a, and generates second feature points in response to the first image 410a. (720a) can be generated.

Second feature points 720a and 720b generated in response to different input images may be compared. As a result of comparing the second feature points 720a and 720b, the loss function of the second feature point acquisition model may be determined so that the two sets of feature points 720a and 720b are equal to each other. In this way, learning of the second feature point acquisition model can be achieved.

In one embodiment, the second feature point acquisition model 710 generates 2-1 descriptors corresponding to the second feature points 720b in response to the converted first image 440a, and the first image 410a ), 2-2 descriptors corresponding to the second feature points 720a may be generated. The 2-1 descriptors and the second 2-2 descriptors generated in response to different input images may be compared. As a result of comparing the 2-1 descriptors and the second 2-2 descriptors, the loss function of the second feature point acquisition model may be determined so that the two sets of descriptors are equal to each other. In this way, learning of the second feature point acquisition model can be achieved.

In one embodiment, the computing device 100 uses a first learning dataset including a first image and labeling information corresponding to the first image to generate first input data corresponding to the first viewpoint of the photographing device. And the second feature point acquisition model may be trained so that feature points corresponding to each other are output for the second input data corresponding to the second viewpoint. Since this second feature point acquisition model can be learned using the labeling information of the first image, which is obtained using the first feature point acquisition model, viewpoint transformation, and viewpoint-inverse transformation, the learning of the second feature point acquisition model Accuracy can be increased, and the learning process of the second feature point acquisition model can be automated. Accordingly, the second feature point acquisition model can operate as a feature point extraction model that is robust to changes in the viewpoint.

In one embodiment, computing device 100 generates second training data having a second viewpoint different from the first viewpoint by applying a viewpoint transformation to the first training data having the first viewpoint, and generating the first training data having a second viewpoint different from the first viewpoint. Input learning data into the second feature point acquisition model to obtain second feature points corresponding to the first learning data, and input the second learning data into the second feature point acquisition model to obtain third feature points corresponding to the second learning data. The viewpoint of the second learning data in which the third characteristic points are reflected can be restored to the first viewpoint by obtaining the viewpoints and applying viewpoint inverse transformation to the second learning data in which the third characteristic points are reflected. The computing device 100 may learn the second feature point acquisition model based on comparing the position information of the third feature points and the position information of the second feature points in the second learning data restored to the first viewpoint. . Here, the computing device 100 creates a second feature point acquisition model based on a loss function set such that the position information of the third feature points in the second learning data restored to the first viewpoint is the same as the position information of the second feature points. You can learn about. Additionally, the computing device 100 is based on a loss function set such that descriptors corresponding to the third feature points and descriptors corresponding to the second feature points in the second learning data restored to the first viewpoint are the same. , the second feature point acquisition model can be trained.

8 shows a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

A component, module, or unit in the present disclosure includes routines, procedures, programs, components, data structures, etc. that perform a specific task or implement a specific abstract data type. Additionally, one of ordinary skill in the art will understand that the methods presented in this disclosure can be used in uni-processor or multiprocessor computing devices, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, etc. ( It will be fully appreciated that each of these may be implemented with other computer system configurations, including those capable of operating in conjunction with one or more associated devices.

Embodiments described in this disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computing devices typically include a variety of computer-readable media. Computer-readable media can be any medium that can be accessed by a computer, and such computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media.

Computer-readable storage media refers to volatile and non-volatile media, transient and non-transitory media, removable and non-removable, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Includes media. Computer readable storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage. This includes, but is not limited to, a device, or any other medium that can be accessed by a computer and used to store desired information.

A computer-readable transmission medium typically implements computer-readable instructions, data structures, program modules, or other data on a modulated data signal, such as a carrier wave or other transport mechanism. Includes all information delivery media. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal have been set or changed to encode information within the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An example environment 2000 is shown that implements various aspects of the invention, including a computer 2002, which includes a processing unit 2004, a system memory 2006, and a system bus 2008. do. Computer 200 herein may be used interchangeably with computing device. System bus 2008 couples system components, including but not limited to system memory 2006, to processing unit 2004. Processing unit 2004 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing units 2004.

System bus 2008 may be any of several types of bus structures that may further be interconnected to a memory bus, peripheral bus, and local bus using any of a variety of commercial bus architectures. System memory 2006 includes read only memory (ROM) 2010 and random access memory (RAM) 2012. The basic input/output system (BIOS) is stored in non-volatile memory (2010), such as ROM, EPROM, and EEPROM, and is a basic input/output system (BIOS) that helps transfer information between components within the computer (2002), such as during startup. Contains routines. RAM 2012 may also include high-speed RAM, such as static RAM for caching data.

Computer 2002 may also read from or use an internal hard disk drive (HDD) 2014 (e.g., EIDE, SATA), magnetic floppy disk drive (FDD) 2016 (e.g., removable diskette 2018). (for writing to), SSDs, and optical disk drives (2020) (e.g., for reading CD-ROM disks (2022) or for reading from or writing to other high-capacity optical media, such as DVDs). Includes. The hard disk drive 2014, magnetic disk drive 2016, and optical disk drive 2020 are connected to a system bus 2008 by a hard disk drive interface 2024, magnetic disk drive interface 2026, and optical drive interface 2028, respectively. ) can be connected to. The interface 2024 for implementing an external drive includes, for example, at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. For the computer 2002, drive and media correspond to storing any data in a suitable digital format. Although the description of computer-readable storage media above refers to removable optical media such as HDDs, removable magnetic disks, and CDs or DVDs, those skilled in the art will also understand removable optical media such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other types of computer-readable storage media may also be used in the exemplary operating environment and that any such media may contain computer-executable instructions for performing the methods of the invention. .

A number of program modules may be stored in the drive and RAM 2012, including an operating system 2030, one or more application programs 2032, other program modules 2034, and program data 2036. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 2012. It will be appreciated that the invention may be implemented on various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 2002 through one or more wired/wireless input devices, such as a pointing device such as a keyboard 2038 and a mouse 2040. Other input devices (not shown) may include microphones, IR remote controls, joysticks, game pads, stylus pens, touch screens, etc. These and other input devices are connected to the processing unit 2004 through an input device interface 2042, which is often connected to the system bus 2008, but may also include a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, It can be connected by other interfaces, etc.

A monitor 2044 or other type of display device is also connected to system bus 2008 through an interface, such as a video adapter 2046. In addition to the monitor 2044, computers typically include other peripheral output devices (not shown) such as speakers, printers, etc.

Computer 2002 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 2048, via wired and/or wireless communications. Remote computer(s) 2048 may be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other conventional network node, and generally refers to computer 2002. For simplicity, only memory storage device 2050 is shown, although it includes many or all of the components described. The logical connections depicted include wired/wireless connections to a local area network (LAN) 2052 and/or a larger network, such as a wide area network (WAN) 2054. These LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, computer 2002 is connected to local network 2052 through wired and/or wireless communications network interfaces or adapters 2056. Adapter 2056 may facilitate wired or wireless communication to LAN 2052, which also includes a wireless access point installed thereon for communicating with wireless adapter 2056. When used in a WAN networking environment, the computer 2002 may include a modem 2058, connected to a communication server on the WAN 2054, or other means of establishing communication over the WAN 2054, such as via the Internet. has Modem 2058, which may be internal or external and a wired or wireless device, is coupled to system bus 2008 via serial port interface 2042. In a networked environment, program modules described for computer 2002, or portions thereof, may be stored in remote memory/storage device 2050. It will be appreciated that the network connections shown are exemplary and that other means of establishing a communications link between computers may be used.

Computer 1602 may be associated with any wireless device or object deployed and operating in wireless communications, such as a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communications satellite, wirelessly detectable tag. Performs actions to communicate with any device or location and telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of illustrative approaches. It is to be understood that the specific order or hierarchy of steps in processes may be rearranged within the scope of the present disclosure, based on design priorities. The method claims of this disclosure provide elements of the various steps in a sample order but are not meant to be limited to the specific order or hierarchy presented.

Claims (17)

A method performed by a computing device, comprising:
Obtaining a first image and a first depth map from an imaging device;
A plurality of transformed images and a plurality of transformations by applying a viewpoint transformation including a rotation matrix and a translation vector to the first image and the first depth map. Obtaining a plurality of converted 3D data including converted depth maps;
Using a pre-trained artificial intelligence-based first feature point acquisition model, acquiring a plurality of sets of first feature points (interest points) corresponding to the plurality of converted images - among the plurality of converted images One converted image corresponds to one set of first feature points among the plurality of sets of first feature points;
Obtaining a plurality of reconstructed three-dimensional data by applying a viewpoint inverse transformation to the plurality of transformed images including the plurality of transformed depth maps and the plurality of sets of first feature points. step; and
Generating labeling information corresponding to the first image based on sets of a plurality of restored first feature points obtained from each of the plurality of restored 3D data - the plurality of restored 3D data One of the restored three-dimensional data corresponds to one set of restored first feature points among the plurality of sets of restored first feature points;
Including,
method. According to claim 1,
Obtaining a first image and a first depth map from the photographing device includes:
Obtaining the first image, the first depth map, and camera information of the photographing device from the photographing device;
Includes, and
Obtaining the plurality of converted 3D data including the plurality of converted images and the plurality of converted depth maps includes:
Converting the first image having a pixel coordinate system into a normalized coordinate system using the camera information;
Converting the first image having the normalized coordinate system into a three-dimensional camera coordinate system using the obtained depth map; and
obtaining the plurality of transformed 3D data including the plurality of transformed images and the plurality of transformed depth maps by applying a viewpoint transformation to the first image having the camera coordinate system;
Including,
method. According to claim 1,
The viewpoint conversion is,
The first image converted to be based on a three-dimensional camera coordinate system indicating the location of the photographing device and the direction in which the photographing device is viewed is converted to a second coordinate value different from the first coordinate value on the camera coordinate system. Converting to,
method. delete delete According to claim 1,
In the viewpoint transformation, a unique transformation matrix is applied to each pixel of the first image,
method. According to claim 1,
The first image is acquired from the imaging device in an unlabeled state, and
The step of generating labeling information corresponding to the first image includes:
generating the labeling information indicating feature points corresponding to the first image;
Including,
method. According to claim 1,
The first feature point acquisition model is,
Corresponding to a pre-trained model based on supervised learning to recognize as a feature point a point where the amount of change in at least one of the color or geometric pattern in the object exceeds a predetermined threshold,
method. According to claim 1,
The viewpoint inverse transformation is,
A rotation matrix (R-1) corresponding to the inverse of the rotation matrix (R) used in the viewpoint transformation and a translation vector (-R) corresponding to the inverse of the translation vector (T) used in the viewpoint transformation. using -1*T),
method. According to claim 1,
The step of obtaining a plurality of sets of first feature points corresponding to the plurality of converted images is:
Obtaining a plurality of two-dimensional data by converting the plurality of converted 3D data having a 3D camera coordinate system into a normalized coordinate system using the plurality of converted depth maps;
Obtaining the plurality of converted images by converting the plurality of two-dimensional data having the normalized coordinate system into a pixel coordinate system based on camera information of the photographing device obtained from the photographing device; and
Inputting the plurality of converted images into the first feature point acquisition model to obtain a plurality of sets of first feature points corresponding to the plurality of converted images;
Including,
method. According to claim 1,
The steps of acquiring the plurality of restored 3D data are:
Obtaining the plurality of converted images including the plurality of sets of first feature points by combining the plurality of sets of first feature points with the plurality of converted images;
converting the plurality of converted images including the plurality of sets of first feature points into a normalized coordinate system using camera information of the photographing device obtained from the photographing device;
converting the plurality of transformed images including the plurality of sets of first feature points having the normalized coordinate system into a three-dimensional camera coordinate system using the plurality of transformed depth maps; and
Obtaining the plurality of restored 3-dimensional data by applying viewpoint inverse transformation to the plurality of transformed images containing the plurality of sets of first feature points having the 3-dimensional camera coordinate system. steps;
Including,
method. According to claim 1,
Generating labeling information corresponding to the first image based on the sets of the plurality of restored first feature points obtained from each of the plurality of restored 3D data includes:
By applying a plurality of reconstructed depth maps from the plurality of reconstructed 3D data to the plurality of reconstructed 3D data, the plurality of reconstructed 3D data having a 3D camera coordinate system is converted into a 2-dimensional normalized coordinate system. Obtaining a plurality of two-dimensional reconstructed data;
By converting the plurality of two-dimensional reconstructed data into a pixel coordinate system based on applying camera information of the photographing device obtained from the photographing device to the plurality of two-dimensional reconstructed data having the two-dimensional normalized coordinate system, Obtaining a plurality of restored images; and
generating labeling information corresponding to the first image based on the plurality of reconstructed images having the pixel coordinate system;
Including,
method. According to claim 1,
Using a first learning dataset including the first image and labeling information corresponding to the first image, first input data corresponding to the first viewpoint of the photographing device and first input data corresponding to the second viewpoint 2. Learning a second feature point acquisition model to output feature points corresponding to each other for input data;
It further includes, wherein the first viewpoint and the second viewpoint have different viewpoints.
method. According to claim 13,
The step of training the second feature point acquisition model is:
generating second learning data having the second viewpoint by applying the viewpoint transformation to the first training data having the first viewpoint;
Inputting the first learning data into the second feature point acquisition model to obtain second feature points corresponding to the first learning data;
acquiring third feature points corresponding to the second learning data by inputting the second learning data into the second feature point acquisition model;
Restoring the viewpoint of the second learning data in which the third feature points are reflected to the first viewpoint by applying the viewpoint inverse transformation to the second learning data in which the third feature points are reflected; and
Learning the second feature point acquisition model based on comparing the position information of the third feature points in the second learning data restored to the first viewpoint with the position information of the second feature points;
Including,
method. According to claim 14,
The step of learning the second feature point acquisition model is,
Based on a loss function set so that the position information of the third feature points in the second learning data restored to the first viewpoint and the position information of the second feature points are the same,
method. A computer program stored in a computer-readable storage medium, wherein the computer program, when executed by a computing device, causes the computing device to perform the following operations, the operations being:
Obtaining a first image and a first depth map from an imaging device;
By applying a viewpoint transformation including a rotation matrix and a translation vector to the first image and the first depth map, a plurality of transformed three-dimensional data including a plurality of transformed images and a plurality of transformed depth maps are generated. Acquiring action;
An operation of acquiring a plurality of sets of first feature points corresponding to the plurality of converted images using a pre-trained artificial intelligence-based first feature point acquisition model - converting one of the plurality of converted images the image corresponds to one set of first feature points among the plurality of sets of first feature points;
obtaining a plurality of reconstructed three-dimensional data by applying viewpoint inverse transformation to the plurality of transformed images including the plurality of transformed depth maps and the plurality of sets of first feature points; and
An operation of generating labeling information corresponding to the first image based on a plurality of sets of restored first feature points obtained from each of the plurality of restored 3D data - one of the plurality of restored 3D data The restored 3D data corresponds to one set of restored first feature points among the plurality of sets of restored first feature points;
Including,
A computer program stored on a computer-readable storage medium. As a computing device,
at least one processor; and
Memory;
Includes,
The at least one processor:
Obtaining a first image and a first depth map from an imaging device;
By applying a viewpoint transformation including a rotation matrix and a translation vector to the first image and the first depth map, a plurality of transformed three-dimensional data including a plurality of transformed images and a plurality of transformed depth maps are generated. Acquiring action;
An operation of acquiring a plurality of sets of first feature points corresponding to the plurality of converted images using a pre-trained artificial intelligence-based first feature point acquisition model - converting one of the plurality of converted images the image corresponds to one set of first feature points among the plurality of sets of first feature points;
obtaining a plurality of reconstructed three-dimensional data by applying viewpoint inverse transformation to the plurality of transformed images including the plurality of transformed depth maps and the plurality of sets of first feature points; and
An operation of generating labeling information corresponding to the first image based on a plurality of sets of restored first feature points obtained from each of the plurality of restored 3D data - one of the plurality of restored 3D data The restored 3D data corresponds to one set of restored first feature points among the plurality of sets of restored first feature points;
To perform,
Computing device.

Images