KR102321998B1 - Method and system for estimating position and direction of image - Google Patents

Abstract

Disclosed are a method and system for estimating the position and orientation of an image that is robust to environmental changes. The image-based positioning method includes: synthesizing a query image according to a positioning request through an image synthesis model; selecting a reference image for positioning using the query image and the synthesized image; and estimating a pose for the query image using the feature points extracted from the query image and the synthesized image and the feature points extracted from the reference image.

Description

Method and system for estimating the position and orientation of an image that is robust to environmental changes

The description below relates to image-based positioning technology.

Various services based on location are provided to wireless terminals. In other words, location-based services such as showing a map, providing a navigation function, or tracking a location based on a location where a wireless terminal is located are being provided.

Accurate positioning technology is required to provide location-based services indoors and outdoors. For example, Korean Patent Publication No. 10-0723680 (registration date of May 23, 2007) discloses a technique for measuring the location of a mobile communication terminal using GPS in an indoor/outdoor environment.

Positioning technology using GPS analyzes a GPS signal received from a GPS satellite to calculate the current position, but the accuracy is not high.

Accordingly, as a high-accuracy positioning technique is required, research and development for measuring a position based on an image is being actively conducted. However, the image-based positioning technology is sensitive to various environmental changes that may occur in an image, such as moving objects or people, weather, illuminance, and time.

A method and system for image-based positioning that is robust to various environmental changes are provided.

We provide a method and system for estimating an accurate 6-degrees of freedom (6-DOF) pose based on an image.

An image-based positioning method executed on a computer system, the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory, the image-based positioning method being performed by the at least one processor , synthesizing a query image according to a location request through an image synthesis model; selecting, by the at least one processor, a reference image for positioning using the query image and the synthesized image; and estimating, by the at least one processor, a pose for the query image by using the feature points extracted from the query image and the synthesized image and the feature points extracted from the reference image. to provide.

According to an aspect, the synthesizing may generate a composite image through image translation using the image synthesis model.

According to another aspect, the synthesizing may include synthesizing the query image with a generative adversarial network (GAN) model that converts a night image into a day image.

According to another aspect, the GAN model may be configured as a network that transforms at least one feature among a color, a texture, and a gradient of an image.

According to another aspect, the selecting may include: searching for at least one first candidate image matching the query image using a global feature extracted from the query image; searching for at least one second candidate image matching the synthesized image using global features extracted from the synthesized image; and selecting at least one of the first candidate image and the second candidate image as the reference image.

According to another aspect, the selecting of at least one of the first candidate image and the second candidate image as the reference image includes the first candidate image based on a matching score of the query image or the synthesized image. and some candidate images among the second candidate images may be selected as the reference image.

According to another aspect, the selecting may further include excluding an image in which time information related to an image from among the first candidate image and the second candidate image has elapsed for more than a predetermined time from being selected as the reference image. can do.

According to another aspect, the estimating may include: acquiring information about visual correspondences between a feature point of the query image or the synthesized image and a feature point of the reference image; and calculating a 6-degrees of freedom (6-DOF) pose corresponding to the query image based on the visual relevance information.

According to another aspect, the calculating of the 6-DOF pose includes a re-projection error (re) in which an error between the feature points is calculated based on the relation between the feature points of the query image or the synthesized image and the feature points of the reference image. The 6-DOF pose may be calculated using a -projection error) measurement method.

There is provided a computer program stored in a non-transitory computer-readable recording medium for executing the image-based positioning method in the computer system.

There is provided a non-transitory computer-readable recording medium in which a program for executing the image-based positioning method in a computer is recorded.

A computer system comprising: at least one processor configured to execute computer readable instructions contained in a memory, the at least one processor comprising: an image synthesizing unit for synthesizing a query image according to a positioning request through an image synthesizing model; a candidate selection unit for selecting a reference image for positioning using the query image and the synthesized image; and a pose estimator for estimating a pose for the query image using the feature points extracted from the query image and the synthesized image and the feature points extracted from the reference image.

According to embodiments of the present invention, it is possible to provide an accurate positioning result by synthesizing an image robust to environmental changes and using the synthesized image for positioning.

According to embodiments of the present invention, an accurate 6-DOF pose can be estimated by synthesizing an image robust to environmental changes and then performing a feature point-based pose estimation using the synthesized image.

1 is a block diagram illustrating an example of an internal configuration of a computer system according to an embodiment of the present invention.
2 is a diagram illustrating an example of components that a processor of a computer system according to an embodiment of the present invention may include.
3 is a flowchart illustrating an example of an image-based positioning method that a computer system can perform according to an embodiment of the present invention.
4 to 5 show examples of GAN models for image synthesis according to an embodiment of the present invention.
6 is an exemplary view for explaining a process of synthesizing a night image into a day image according to an embodiment of the present invention.
7 illustrates an example of a process of selecting a candidate image as a reference image for positioning according to an embodiment of the present invention.
8 to 10 show an example of a process for estimating a 6-DOF pose according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Embodiments of the present invention relate to a technique for estimating the position and orientation of an image that is robust to environmental changes.

Embodiments including those specifically disclosed in this specification can estimate an accurate 6-DOF pose by using an image that is robust to environmental changes, and through this, various aspects such as accuracy, precision, and optimization of a positioning technique using an image achieve significant advantages in

1 is a block diagram illustrating an example of a computer system according to an embodiment of the present invention. For example, the image-based positioning system according to embodiments of the present invention may be implemented by the computer system 100 illustrated in FIG. 1 .

As shown in FIG. 1 , the computer system 100 is a component for executing the image-based positioning method according to embodiments of the present invention, and includes a memory 110 , a processor 120 , a communication interface 130 , and input/output. It may include an interface 140 .

The memory 110 is a computer-readable recording medium and may include a random access memory (RAM), a read only memory (ROM), and a permanent mass storage device such as a disk drive. Here, a non-volatile mass storage device such as a ROM and a disk drive may be included in the computer system 100 as a separate permanent storage device distinct from the memory 110 . Also, an operating system and at least one program code may be stored in the memory 110 . These software components may be loaded into the memory 110 from a computer-readable recording medium separate from the memory 110 . The separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, the software components may be loaded into the memory 110 through the communication interface 130 rather than the computer-readable recording medium. For example, the software components may be loaded into the memory 110 of the computer system 100 based on a computer program installed by files received over the network 160 .

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 120 by the memory 110 or the communication interface 130 . For example, the processor 120 may be configured to execute a received instruction according to a program code stored in a recording device such as the memory 110 .

The communication interface 130 may provide a function for the computer system 100 to communicate with other devices via the network 160 . For example, a request, command, data, file, etc. generated by the processor 120 of the computer system 100 according to a program code stored in a recording device such as the memory 110 is transmitted to the network ( 160) to other devices. Conversely, signals, commands, data, files, etc. from other devices may be received by the computer system 100 through the communication interface 130 of the computer system 100 via the network 160 . A signal, command, or data received through the communication interface 130 may be transferred to the processor 120 or the memory 110 , and the file may be a storage medium (described above) that the computer system 100 may further include. persistent storage).

The communication method is not limited, and a communication method using a communication network (eg, a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network 160 may include may also include short-distance wired/wireless communication between devices. have. For example, the network 160 may include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), and a broadband network (BBN). , the Internet, and the like. In addition, the network 160 may include any one or more of a network topology including a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc. not limited

The input/output interface 140 may be a means for an interface with the input/output device 150 . For example, the input device may include a device such as a microphone, keyboard, camera, or mouse, and the output device may include a device such as a display or speaker. As another example, the input/output interface 140 may be a means for an interface with a device in which functions for input and output are integrated into one, such as a touch screen. The input/output device 150 may be configured as one device with the computer system 100 .

Also, in other embodiments, computer system 100 may include fewer or more components than those of FIG. 1 . However, there is no need to clearly show most of the prior art components. For example, the computer system 100 is implemented to include at least a portion of the above-described input/output device 150 or other components such as a transceiver, a global positioning system (GPS) module, a camera, various sensors, and a database. may include more. As a more specific example, the computer system 100 may be implemented in the form of a server system capable of estimating the position and direction of an image based on an image received from a wireless terminal or a mobile robot, and in general, various components included in the server system They may be implemented to be further included in the computer system 100 .

The present invention relates to an image-based localization (visual localization) technology, and is applicable to both indoor and outdoor localization.

FIG. 2 is a diagram illustrating an example of components that a processor of a computer system according to an embodiment of the present invention may include, and FIG. 3 is an image-based view that the computer system according to an embodiment of the present invention may include. It is a flowchart showing an example of a positioning method.

As shown in FIG. 2 , the processor 120 may include an image synthesis unit 201 , a candidate selection unit 202 , and a pose estimation unit 203 . These components of the processor 120 may be representations of different functions performed by the processor 120 according to a control instruction provided by at least one program code. For example, the image synthesizing unit 201 may be used as a functional representation that operates to control the computer system 100 so that the processor 120 synthesizes an image robust to environmental changes.

The processor 120 and the components of the processor 120 may perform steps S310 to S340 included in the image-based positioning method of FIG. 3 . For example, the processor 120 and components of the processor 120 may be implemented to execute an operating system code included in the memory 110 and an instruction according to at least one program code described above. Here, at least one program code may correspond to a code of a program implemented to process the image-based positioning method.

The image-based positioning method may not occur in the order shown, and some of the steps may be omitted or additional processes may be further included.

The processor 120 may load the program code stored in the program file for the image-based positioning method into the memory 110 . For example, the program file for the image-based positioning method may be stored in a persistent storage device separate from the memory 110, and the processor 120 stores the program code from the program file stored in the persistent storage device through the bus to the memory. The computer system 100 may be controlled to be loaded into 110 . At this time, each of the processor 120 and the image synthesizing unit 201 , the candidate selecting unit 202 , and the pose estimating unit 203 included in the processor 120 corresponds to a corresponding portion of the program code loaded into the memory 110 . may be different functional representations of the processor 120 for executing the subsequent steps S310 to S340 by executing the instruction of . For the execution of steps S310 to S340 , the processor 120 and components of the processor 120 may directly process an operation according to a control command or control the computer system 100 .

In step S310 , when a query image according to a location request is received, the image synthesizing unit 201 may synthesize the query image into an image that is robust to environmental changes (synthesize). For example, the image synthesizing unit 201 may generate an image robust to environmental changes (hereinafter, referred to as a 'synthetic image') through an image synthesizing model (eg, a generative adversarial network (GAN)) with respect to the query image. The image synthesizing unit 201 may input an image taken at night (hereinafter, referred to as a 'night image') as a query image into the image synthesis model and synthesized into an image taken during the day (hereinafter, referred to as a 'day image'). .

In step S320 , the candidate selector 202 is a query image and a reference image for the image synthesized in step S310 , which is a query image in the map DB (the above-mentioned permanent storage device) for VL (visual localization). and a candidate image highly related to the composite image may be selected. For example, the candidate selector 202 extracts global features from the query image and the composite image and compares them with global features of each image on the VL map DB to select an image with a high matching score as a candidate image. have.

In operation S330 , the pose estimator 203 may extract information on visual correspondences between the query image and the composite image with the candidate image selected in operation S320 . For example, the pose estimator 203 may extract feature points that are local features from the query image, the composite image, and the candidate image, respectively, and then extract visual relevance information between the feature points of the corresponding images.

In step S340 , the pose estimator 203 may estimate a 6-DOF pose corresponding to the query image as a positioning result for the query image based on the visual relevance information extracted in step S330 . For example, the pose estimator 203 may calculate the 6-DOF pose using a re-projection error measurement method, which is a method of calculating an error between the feature points based on the feature points that are local features of the image.

A specific example of the process of synthesizing the query image into an image that is robust to environmental changes will be described as follows.

image-to-image translation

GAN is an image-to-image transformation approach that produces a conditioned output from a given input.

The CycleGAN model, which is a GAN model that extends image-to-image transformation to an unsupervised framework, does not require alignment of image pairs. An adversarial training model includes a generator and a discriminator network, where a discriminator D trains to distinguish one or more real and generated sample pairs, and a generator G uses the generated samples. Train to deceive discriminator D.

4 shows an exemplary diagram for explaining the CycleGAN model in an embodiment of the present invention.

a, G_A(G_B(b))

b와 같이 G_A와 G_B가 반전되도록 유도함으로써 이미지를 단방향으로 변환하는 문제를 규칙화할 수 있다.Referring to FIG. 4 , the CycleGAN model 40 may include two pairs of a generator 41 and a discriminator 42 _{(G A} , D _A ) and (G _B , D _{B ).} Here, the translators between domains A and B are G _A : A→B and G _B : B→A. D _A is trained to distinguish between the real image a and the transformed image G _B (b), and D _B is trained to distinguish between the images b and G _A (a). The CycleGAN model 40 is trained using both an adversarial loss and a cycle loss. Periodic coherence loss is G _B (G _A (a)

a, G _A (G _B (b))

_{By inducing G A} and G _B to be inverted as in b, we can regularize the problem of transforming the image in one direction.

The target of the CycleGAN model 40 may be expressed as in Equations 1 to 3.

[Equation 1]

[Equation 2]

[Equation 3]

place awareness and localization

Location recognition is location classification and refers to an operation of identifying an actual location in an image of a specific location.

Image-based localization (VL) is the process of identifying a camera position (and orientation) by comparing it to a local or global feature map. One method for image-based localization (VL) is to find the most similar image among images tagged with a pose for a query through an image search. In this case, changing the camera pose is not desirable because the camera pose plays a very important role in the pose calculation.

image-based positioning ( VL ) for image translation

In the embodiment of the present invention, a GAN model that deals with the localization problem between a set of nighttime images, which is a query image, and a set of pose-tagged daytime images, is used. The GAN model trains the transformation between the daytime image domain and the nighttime image domain, and applies a night-to-day direction to transform the night image into the day image. In this case, both the transformed image and the reference image may obtain a feature vector per image. The nearest neighbor search provides a reference image that best matches the query image, and the pose of the query image is approximated by the pose of the reference image.

In the present invention, an image synthesis model modified by the CycleGAN model 40 may be applied to synthesize a query image into an image that is robust to environmental changes.

The generator network of the image synthesis model is the same as that of the generator 41 used in the CycleGAN model 40, but by dividing it in half, the former generator may be configured as an encoder and the latter generator as a decoder. For both domains, the structure and training procedure of the generator network are the same as those of the CycleGAN model (40).

The discriminator network can be implemented by modifying the discriminator 42 used in the CycleGAN model 40 .

5 shows an example of a discriminator network structure for image transformation according to an embodiment of the present invention.

5, in the discriminator network of the image synthesis model according to the present invention, the discriminator 52 of each domain is composed of three network clones (blur-RGB, gray scale, xy-gradient) for the input image. can Each discriminator 52 outputs a decision that can cover different receptive-field sizes in the convolution layer.

One of the three network clones takes the RGB image 502 in which the input image 501 is blurred through a 5×5 3σ Gaussian kernel, the other takes the luminance data 503 of the input image 501, and The other takes the horizontal/vertical gradient data 504 of the input image 501 . The discriminator 52 of this configuration may focus on a color, a texture, and a gradient through each network clone, respectively. Each network clone of the discriminator is identical in architecture and hyperparameters, and the losses are averaged and eventually identical.

A network clone taking the horizontal/vertical gradient data 504 serves to mimic the process of extracting a scale invariant feature transform (SIFT) descriptor. After converting the input image 501 to gray scale, all other pixels are skipped and downsampled by a factor of two. Then, the gradient is computed using the [-1 0 1] kernel for the x-direction gradient and the y-direction transpose, and then a histogram of the magnitude-weighted gradient is generated.

Therefore, the image synthesis model according to the present invention can obtain a downsampled image using 1×1 stride-2 convolution and combine with two filters to obtain the same xy-gradient in different ways for the discriminator. The structural discriminator 52 above can be used to create relevant features that match the transformed version that do not exist in the original, while retaining relevant features of the original image in cyclic reconstruction.

And, it is possible to include the gradient absolute value and direction in the discriminator 52 and output a label/decision after each downsampling layer. Differentiating images on multiple scales allows consistency for both low-level as well as high-level images rather than arbitrary receptive field sizes.

로 나누어 평균을 산출한다.The image synthesis model according to the present invention outputs multiple crystals for each discriminator 52 . The final loss is linearly weighted in ascending order to the last layer as the complexity and performance of the network prediction increases with depth. In ascending hierarchical order [1, 2, … , n] can be seen as output n with weights of

Divide by to get the average.

In the image synthesis model according to the present invention, the loss formula of the discriminator 52 is applied, which is changed to a training form that determines how realistic the input is compared to the fake, rather than determining the realism in an absolute way. This is intended to stabilize training as a whole by preventing the discriminator 52 from becoming too powerful with respect to the constructor 41 .

Equation 4 defines the least squares GAN loss by applying Equation 1 to the relativistic loss formula.

[Equation 4]

The image synthesizing unit 201 may generate an image robust to environmental changes through the image synthesizing model described with reference to FIGS. 4 and 5 . As shown in FIG. 6 , the image synthesizing unit 201 may synthesize a night image 601 input as a query image into a daytime image 602 through the GAN model 60 .

In the above description, the GAN model, which is one of the image synthesis models, is used to synthesize the query image into an image that is robust to environmental changes, but the present invention is not limited thereto, and any technology capable of converting an image can be applied. For example, it is also possible to synthesize an image that is resistant to environmental changes by adjusting the brightness of the query image.

7 illustrates an example of a process of selecting a candidate image as a reference image for positioning according to an embodiment of the present invention.

Referring to FIG. 7 , the candidate selector 202 selects at least one reference image matching the query image 601 which is the original image as the first candidate image group, and is synthesized through the GAN model 60 . At least one reference image matching the image 602 may be selected as the second candidate image group. At this time, the candidate selector 202 extracts global features from the query image 601 and the composite image 602 through a deep learning model, and then uses the extracted features to search for matching images on the VL map DB. can do. For the first candidate image group and the second candidate image group, a predetermined number of images may be selected based on a matching score with the query image 601 or the composite image 602, or images with a predetermined score or higher may be selected. have.

The candidate selector 202 may determine the final reference image 703 using the first candidate image group and the second candidate image group. For example, both the first candidate image group and the second candidate image group may be used as the final reference image 703 . As another example, by arranging the first candidate image group and the second candidate image group based on the matching score, only some candidate images having a high matching score may be used as the final reference image 703 .

In this case, the candidate selector 202 selects an image in which time information (eg, image creation time, image update time, etc.) related to the images in the first candidate image group and the second candidate image group has elapsed for a certain period of time or more as a final reference image ( 703) can be excluded.

Accordingly, the candidate selector 202 may select the reference image 703 by using not only the query image 601 but also the composite image 602 .

8 illustrates an example of a process for estimating a 6-DOF pose according to an embodiment of the present invention.

Referring to FIG. 8 , the pose estimator 203 extracts local features (2D feature points) from the query image 601 and the composite image 602 as well as local features (3D feature points) from the final reference image 703. can be extracted. For example, the pose estimator 203 may include a Scale Invariant Feature Transform (SIFT), Speeded up robust features (SURF), Features from Accelerated Segment Test (FAST), Binary robust independent elementary features (BRIEF), Oriented FAST and ORB (Oriented FAST and Rotated BRIEF) can extract key points that are visual features from the image through a key point extraction algorithm.

Thereafter, the pose estimator 203 uses descriptor information of the feature points extracted from the query image 601 and the composite image 602 and the feature points extracted from the final reference image 703 to correlate the feature points between different images. ) can be obtained. As shown in FIG. 9 , the pose estimator 203 performs two different images 601 and 703 from the feature point matching result of the query image 601 (or the composite image 602 ) and the final reference image 703 . relationship between them can be obtained.

In addition, the pose estimator 203 may calculate a 6-DOF pose using a re-projection error measuring method, which is a method of calculating an error between the feature points based on feature points that are local features of two different images 601 and 703 . .

)를 수학식 5와 같이 정의할 수 있다.Referring to FIG. 10 , a re-projection error (re-projection error) using the association obtained between different images

) can be defined as in Equation 5.

[Equation 5]

)(

)

는 투영 함수(projection function)이고,

는 6자유도 포즈를 나타내는 4×4 변환 행렬(transformation matrix)이다. 그리고,

는 최종 참조 이미지(703)의 특징점의 3차원 포인트이고,

는 쿼리 이미지(601) 또는 합성 이미지(602)의 특징점의 2차원 픽셀 좌표이다.here,

is the projection function,

is a 4×4 transformation matrix representing a 6-DOF pose. and,

is the three-dimensional point of the feature point of the final reference image 703,

is the two-dimensional pixel coordinates of the feature points of the query image 601 or the composite image 602 .

)를 얻을 수 있다.When the errors are summed for all relevance, an energy function (energy function) in the form of Equation 6

) can be obtained.

[Equation 6]

)가 최소화되는 방향으로 변환 행렬인

를 계산할 수 있다. 이때, 변환 행렬(

)의 계산을 위해서는 수학식 7과 같은 screw displacement 기법을 사용할 수 있다.The above-mentioned energy function (

) is the transformation matrix in the direction in which it is minimized.

can be calculated. In this case, the transformation matrix (

) can be calculated using the screw displacement technique as in Equation 7.

[Equation 7]

)(

)

은 회전 행력(rotation matrix)이고,

는 병진 벡터(translation vector)이다. 그리고,

는 트위스트 벡터(twist vector)를 의미하고,

,

,

는 각 축의 회전 속도(rotation velocity)를 의미하고,

,

,

는 각 축의 병진 속도를 의미한다.here,

is the rotation matrix,

is a translation vector. and,

means a twist vector,

,

,

is the rotation velocity of each axis,

,

,

is the translational speed of each axis.

)가 최소화되는 방향으로 6-DOF 포즈인

를 계산할 수 있고, 이때 가우스-뉴턴(Gauss-Newton) 기반의 비선형 최적화(non-linear optimization) 기법을 활용하여 점진적으로 포즈를 계산할 수 있다.The pose estimator 203 is an energy function (

) is a 6-DOF pose in the direction where it is minimized.

can be calculated, and at this time, the pose can be gradually calculated using a Gauss-Newton-based non-linear optimization technique.

Accordingly, the pose estimator 203 may calculate a 6-DOF pose for the query image 601 based on the correlation information of the feature points obtained between different images through the above-described process.

As described above, according to embodiments of the present invention, an accurate reference image can be extracted using the synthesized image by synthesizing an image robust to environmental changes, and an accurate positioning result can be provided. And, according to embodiments of the present invention, an accurate 6-DOF pose can be estimated by synthesizing an image robust to environmental changes and then performing a feature point-based pose estimation using the synthesized image together with the query image.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the apparatus and components described in the embodiments may include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable logic unit (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit, microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be embodied in any type of machine, component, physical device, computer storage medium or device for interpretation by or providing instructions or data to the processing device. have. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be to continuously store a program executable by a computer, or to temporarily store it for execution or download. In addition, the medium may be a variety of recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (20)

An image-based positioning method executed on a computer system, comprising:
the computer system comprising at least one processor configured to execute computer readable instructions contained in a memory;
The image-based positioning method,
synthesizing, by the at least one processor, a query image according to a location request through an image synthesis model;
selecting, by the at least one processor, a reference image for positioning using the query image and the synthesized image; and
estimating, by the at least one processor, a pose for the query image using the feature points extracted from the query image and the synthesized image and the feature points extracted from the reference image;
including,
The selecting step is
searching for at least one first candidate image matching the query image using a global feature extracted from the query image;
searching for at least one second candidate image matching the synthesized image using global features extracted from the synthesized image; and
selecting at least one of the first candidate image and the second candidate image as the reference image;
An image-based positioning method that includes. According to claim 1,
The synthesizing step is
Generating a composite image through image translation using the image synthesis model
An image-based positioning method, characterized in that According to claim 1,
The synthesizing step is
synthesizing the query image through a generative adversarial network (GAN) model that converts a night image into a day image
An image-based positioning method, characterized in that 4. The method of claim 3,
The GAN model is composed of a network that transforms at least one feature of an image color, texture, and gradient
An image-based positioning method, characterized in that delete According to claim 1,
selecting at least one of the first candidate image and the second candidate image as the reference image includes:
Selecting some candidate images from among the first candidate image and the second candidate image as the reference image based on a matching score of the query image or the synthesized image
An image-based positioning method, characterized in that According to claim 1,
The selecting step is
excluding an image in which time information related to an image has elapsed for more than a predetermined time from among the first candidate image and the second candidate image from being selected as the reference image;
An image-based positioning method further comprising a. According to claim 1,
The estimating step is
obtaining visual correspondences information between the feature points of the query image or the synthesized image and the feature points of the reference image; and
calculating a 6-degrees of freedom (6-degrees of freedom) pose corresponding to the query image based on the visual relevance information
An image-based positioning method comprising a. 9. The method of claim 8,
Calculating the 6-DOF pose comprises:
Calculating the 6-DOF pose using a re-projection error measuring method, which is a method of calculating an error between feature points based on the relation between the feature points of the query image or the synthesized image and the feature points of the reference image thing
An image-based positioning method, characterized in that A computer program stored in a non-transitory computer-readable recording medium for executing the image-based positioning method of any one of claims 1 to 4, 6 to 9 in the computer system. A non-transitory computer-readable recording medium having a program recorded thereon for executing the image-based positioning method of any one of claims 1 to 4 and 6 to 9 on a computer. In a computer system,
at least one processor configured to execute computer readable instructions contained in memory
including,
the at least one processor,
an image synthesizing unit for synthesizing a query image according to a positioning request through an image synthesizing model;
a candidate selection unit for selecting a reference image for positioning using the query image and the synthesized image; and
A pose estimator for estimating a pose for the query image using the feature points extracted from the query image and the synthesized image and the feature points extracted from the reference image
including,
The candidate selection unit
searching for at least one first candidate image matching the query image using the global feature extracted from the query image,
searching for at least one second candidate image matching the synthesized image using the global features extracted from the synthesized image;
selecting at least one of the first candidate image and the second candidate image as the reference image
A computer system characterized by a. 13. The method of claim 12,
The image synthesizing unit,
Generating a composite image through image conversion using the image synthesis model
A computer system characterized by a. 13. The method of claim 12,
The image synthesizing unit,
Compositing the query image through a GAN model that converts a night image into a day image
A computer system characterized by a. 15. The method of claim 14,
The GAN model consists of a network that transforms at least one feature among color, texture, and gradient of an image.
A computer system characterized by a. delete 13. The method of claim 12,
The candidate selection unit
Selecting some candidate images from among the first candidate image and the second candidate image as the reference image based on a matching score of the query image or the synthesized image
A computer system characterized by a. 13. The method of claim 12,
The candidate selection unit
Excluding an image in which time information related to an image has elapsed for a predetermined time or more from among the first candidate image and the second candidate image is not selected as the reference image
A computer system characterized by a. 13. The method of claim 12,
The pose estimation unit,
Obtaining visual relevance information between the feature points of the query image or the synthesized image and the feature points of the reference image,
Calculating a 6-DOF pose corresponding to the query image based on the visual relevance information
A computer system characterized by a. 20. The method of claim 19,
The pose estimation unit,
Calculating the 6-DOF pose using a re-projection error measuring method, which is a method of calculating an error between the feature points based on the relation between the feature points of the query image or the synthesized image and the feature points of the reference image
A computer system characterized by a.

Images