KR20200056708A - Method and apparatus for location determination using image processing and location information - Google Patents

Abstract

According to one embodiment of the present invention, provided are a method and an apparatus for determining a location using image processing and location information, which comprises the steps of: receiving a target image which photographed a specific photographing location from an image photographing device; extracting at least one feature point from the target image; selecting at least one matching image from a plurality of images based on the at least one feature point and feature points of the plurality of pre-stored images; and determining a final position using location information included in each of the target image and the at least one matching image and the matching similarity between the images based on the feature point.

Description

Positioning method and device using image processing and location information {METHOD AND APPARATUS FOR LOCATION DETERMINATION USING IMAGE PROCESSING AND LOCATION INFORMATION}

The present invention relates to a method and apparatus for positioning using image processing and location information, and more particularly, to a method and apparatus for determining a more accurate location using matching points and location information of an image.

Photography is a very efficient tool for leaving memories, and with the spread of smartphones, anyone can easily leave pictures. In particular, since photographs on travel destinations have a special meaning, many people take photos at places where they can represent each travel destination.

However, many people do not know the geography of the destination and often miss the sights. To this end, a system that recommends some nearby spots that many people who took photos at the current location after grasping the current location based on the location information (GPS, Global Positioning System) and photo information of the current user's location This may be necessary. In order to increase the reliability of place recommendation, it is necessary to accurately locate the current user. There are two methods: image processing and GPS, but the image processing has the disadvantage of being less reliable in distinguishing between points with similar images, and in the case of GPS, it is difficult to distinguish only with GPS in indoors. Therefore, in order to compensate each other's shortcomings, a complex model combining two pieces of information is required.

Republic of Korea Patent Publication No. 10-2017-0052674 (2017. 05. 12)

An object of the present invention is to provide a method and apparatus capable of determining a more accurate location using image information and location information of an image.

In addition, it is to provide a method and apparatus for informing a user of nearby attractions and places based on a more accurate location.

The problem to be solved by the present invention is not limited to the problem (s) mentioned above, and another problem (s) not mentioned will be clearly understood by those skilled in the art from the following description.

A method of positioning using image processing and location information according to an embodiment of the present invention includes receiving a target image photographing a specific photographing location from an image photographing apparatus, and extracting at least one feature point from the target image, Selecting at least one matching image among the plurality of images based on the at least one feature point and the feature points of a plurality of pre-stored images, and location information and the feature points included in each of the target image and the at least one matching image And determining a final position of the target image using matching similarity between images based on.

Preferably, the feature point extraction may use a scale-invariant feature transform (SIFT), speeded up robust features (SURF), or an Oriented FAST and Rotated BRIEF (ORB) feature point extraction algorithm.

Preferably, the step of selecting the matching image may be performed by feature matching between comparison images.

Preferably, the step of determining the final position of the target image using the matching similarity between the images may use Equation 1, Equation 2, and Equation 3.

[Equation 1]

I _Place (C) = MAX (I _V (P _i , C))

Here, I _Place (C) is the highest matching similarity, I _V (P _i , C) is the matching similarity between the at least one matching image and the target image, C is the target image, P _i is the i-th This is the matching image.

[Equation 2]

I _V (P _i , C) = wD (G (P _i ), G (C)) + (1-w) L _i (however 0 <w <1)

Here, w is a weight, D (G (P _i ), G (C)) is G (C) is location information of the target image, G (P _i ) is i-th matching image location information, D (G ( P _i ), G (C)) is the Euclidean distance between the location of the target image and the location of the matching image, and L _i is a softmax function for the number of optimal matching points of P _i .

[Equation 3]

Here, i is the i-th matching image, n _pl is the plurality of _pre -stored images, j is the j-th matching image among the pre-stored plurality of images, and f is the number of optimal matching points of the i or j-th matching image, respectively. .

Preferably, the method may further include recommending at least one location based on the final location of the target image.

Preferably, the step of recommending the at least one location may be recommended using Equation 4 and Equation 5 below.

[Equation 4]

Order = freq * C ^{(distance * gain)} (however, 0 <gain <1)

Here, Order is the recommended score of the recommended place, freq is the frequency of visits to the recommended places of multiple users, distance is the distance between the final location of the target image and the location of the recommended place, and gain is weight.

[Equation 5]

및

은 상기 대상이미지의 최종위치의 위도 및 경도,

및

는 추천된 장소의 위도 및 경도이다.here,

And

Is the latitude and longitude of the final position of the target image,

And

Is the latitude and longitude of the recommended place.

In addition, the positioning apparatus using image processing and location information includes: a receiving unit that receives a target image photographing a specific shooting location from an image photographing apparatus, an extraction unit that extracts at least one feature point from the target image, and the at least one feature point A selection unit for selecting at least one matching image among the plurality of images based on the feature points of the previously stored images, location information included in each of the target image and the at least one matching image, and an image based on the feature point It includes a determining unit for determining the final position of the target image using the matching similarity between.

Preferably, a recommendation unit for recommending at least one location based on the final location of the target image may be further included.

The present invention has an effect of determining an accurate location without distinction between indoor and outdoor by varying weights for image information and location information of an image.

In addition, the present invention has an effect of providing a recommendation place around the user to the user based on the exact location.

1 is a flowchart of a method for positioning using image processing and location information according to an embodiment of the present invention.
2 is a view showing a matching image according to an embodiment of the present invention.
3 to 5 are diagrams showing experimental results according to an embodiment of the present invention.
6 is a view showing a target image according to an embodiment of the present invention.
7 is a view showing a change in probability according to a weight change according to an embodiment of the present invention.
8 is a block diagram illustrating a positioning device according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term and / or includes a combination of a plurality of related described items or any one of a plurality of related described items.

When an element is said to be "connected" or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but there may be other components in between. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, one or more other features. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and should not be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. Does not.

1 is a flowchart of a method for positioning using image processing and location information according to an embodiment of the present invention.

Referring to FIG. 1, in step S110, the positioning device 800 receives a target image photographing a specific photographing location from an image photographing apparatus.

The video recording device may represent a dedicated video recording device such as a digital camera or camcorder, and a mobile phone or a convergence device equipped with a camera or camcorder function in a portable device such as a mobile phone or a game machine, but in the present invention, GPS location information is received. It is preferable to interpret the location of the photographed image as a camera of a smart device that can be stored together with the target image (hereinafter referred to as an image photographing apparatus).

Accordingly, the positioning device 800 can receive and store a target image including information on the location from the image photographing device.

Here, the location information included in the target image may not indicate the exact location depending on external conditions (weather, radio interference, etc.) or internal conditions of the image photographing device. More accurate through the positioning device 800 proposed by the present invention It is an object of the present invention to provide a location.

In step S120, the positioning device 800 extracts at least one feature point from the target image.

When the positioning device 800 tracks or recognizes an object in an image, the most common method when matching an image to an image is to extract and match a key key point in the image.

As a condition of these feature points, it should be easy to identify even if the shape, size, or position of an object changes, or even if the camera's point of view or lighting changes, it should be easy to find the point in the image. The best feature point that satisfies this condition in the image is the corner point. And most feature point extraction algorithms are based on such corner point detection, but the present invention is not limited thereto.

The method for the positioning device 800 to extract the feature points on the image may be extracted using Harris corner, Harris corner, SIFT-DoG, FAST, AGAST, or the like.

In another embodiment, the feature extraction of the positioning device 800 may use a scale-invariant feature transform (SIFT), speeded up robust features (SURF), or Oriented FAST and Rotated BRIEF (ORB) feature extraction algorithm.

For example, in the case of SIFT, a feature point is selected from corners and edges in an image, and a histogram for gradient direction and size within a certain block size area is obtained around it. Then, feature information representing each bin value as a vector of a certain dimension (eg, 128 dimensions) can be generated.

In the present invention, it is needless to say that various main constant characteristic methods such as BRIEF, FREAK, etc. other than SIFT, SURF, and ORB can be used.

In the present invention, the feature points were extracted and tested using SIFT, SURF, and ORB feature point extraction algorithms to simultaneously secure the reaction speed with precision.

In step S130, the positioning device 800 selects at least one matching image among the plurality of images based on at least one feature point and feature points of a plurality of pre-stored images.

A separate database DB is provided in the positioning device 800 to store a plurality of images in advance, and each of the plurality of images may include information (for example, latitude and longitude) corresponding to a location corresponding to each image. have.

As another embodiment, the step of selecting the matching image by the positioning device 800 may be performed by feature matching between comparison images.

For example, the positioning device 800 may select a plurality of matching images having a similarity level higher than a predetermined criterion by comparing the target image and a plurality of images previously stored in the database using a K-NN matching algorithm. Hereinafter, the present invention will be described through specific examples.

The positioning device 800 previously stored and used GPS location information, keypoints, and descriptors of a plurality of images in a database (DB) using MySQL, and used 18 points in Chungnam National University. Did. All tests were conducted on images of coordinates of latitude 36.369443 and longitude 127.344408.

The program used openCV3.4.1 based on python3.5.4 and worked with 'spyder'.

First, the positioning device 800 extracts locations matching the latitude and longitude within ± 0.002 degrees from the DB based on the current user's GPS location information and extracts a matching image. After the positioning device 800 extracted feature points and descriptors from the matching image, a K-NN matching algorithm was used as a matching algorithm to check whether it matches the matching image on the DB. The algorithm is set to find k matches for each feature point and to find 2 matches for testing. In order to use only those with high reliability among these sets of matching points, a ratio test was performed on the sizes of the two matching points, and only the case where the ratio was less than 0.75 was designated as “good match”. When the ratio of the optimal coincidence points did not exceed 20% of the total coincidence points, the image was regarded as not fully recognized. A total of six images were selected as matching images.

Referring to FIG. 2, it is defined as place 1, place 2, place 3, place 4, place 5, and place 6 clockwise from the top left.

The time required for each algorithm was measured 10 times, and as a result, it was confirmed that ORB (0.2565 seconds), SURF (17.41 seconds), and SIFT (84.50 seconds) were fast on average. In order to check the accuracy of the matching images (places 1 to 6), the number of optimal matching points for each algorithm was checked. Referring to FIGS. 3 to 5, it is possible to check the precision of each algorithm determined based on the difference in the number of optimal matching points.

The accuracy of matching images can be confirmed to be ORB-90.32%, SIFT-93.18%, and SURF-88.12%, and the probability of matching the target image is ORB-90.87%, SIFT-92.7%, and SURF-89.08%. Accordingly, it can be confirmed that SIFT has the highest precision of image discrimination.

In step S140, the positioning device 800 determines the final position of the target image by using a matching similarity between the target image and the image based on the location information and feature points included in each of the at least one matching image.

The positioning device 800 uses GPS location information in addition to location detection based on an image. The Euclidean distance from the matching images selected based on the GPS location information of the location where the target image was taken is obtained.

Here, information on the number of matching points obtained by the Softmax function is synthesized to identify a place having the highest matching similarity.

In another embodiment, the step of determining the final position of the target image using the matching similarity between images by the positioning device 800 uses Equation 1, Equation 2, and Equation 3.

[Equation 1]

I _Place (C) = MAX (I _V (P _i , C))

Here, I _Place (C) is the highest matching similarity, I _V (P _i , C) is the matching similarity between at least one matching image and the target image, C is the target image, and P _i is the i-th matching image.

[Equation 2]

I _V (P _i , C) = wD (G (P _i ), G (C)) + (1-w) L _i (however 0 <w <1)

Here, w is the weight, D (G (P _i ), G (C)) is G (C) is location information of the target image, G (P _i ) is location information of the i-th matching image, D (G (P _i ), G (C)) is the Euclidean distance between the location of the target image and the location of the matching image, and L _i is a softmax function for the number of optimal matching points of P _i .

[Equation 3]

Here, i is the i-th matching image, n _pl is a plurality of _pre -stored images, j is a j-th matching image among the plurality of pre-stored images, and f is information on the number of optimal matching points of the i or j-th matching image, respectively.

Since the range of values of the distance information, the optimal number of matching points, and the information between the target image and the matching image are different, both information was normalized to a value between 0 and 1.

The positioning device 800 may use the two pieces of information complementarily according to the weight w. As an example, the image in Figure 3 has a lot of warped images and composition stored in the actual DB, so it is impossible to know that the image corresponds to 'place 2' only by determining the image. However, when the positioning device 800 is applied, the correct position can be grasped by correction of the distance.

This means that it is difficult to know whether the image taken by the user accurately refers to the point through only image processing, but this can be supplemented when the positioning device 800 is applied.

6 and 7, FIG. 6 shows a change in probability for each place when the weight, w, is changed with respect to FIG. 7, which is the same image.

The smaller the w value, the more the positioning device 800 is affected by image processing, and the larger the w value is, the more it is affected by GPS location information. 6 is a result of testing based on FIG. 7, which is a slightly ambiguous result when only image processing is performed, it can be seen that the probability of other places increased as the value of w decreased. Conversely, as the w value increases, it can be seen that the probability of places close to the actual distance increases.

For this reason, if additional locations are added, the weights for each location must be set appropriately.

As a result, as a result of measuring the time required for each algorithm 10 times, on average, the quick reaction speed can be confirmed in the order of ORB (0.2565 sec), SURF (17.41 sec), and SIFT (84.50 sec), especially in the case of ORB. You can see that it shows the result.

The precision of image discrimination is highest with 93.18% of SIFT. However, it is not suitable for use as the main algorithm because the reaction speed is about 330 times slower than ORB to use only because SIFT is the most precise. Finally, it is most preferable to use an ORB with an average response speed of 0.2565 seconds and an average precision of 90.32% as the main algorithm.

The positioning device 800 may determine which of the distance information and the image information to use more importantly through a change in the weight w. For example, in the case of distinguishing floors in buildings where GPS location information does not have much meaning, the value of w is lowered to increase the weight of image information, and the brightness or state of multiple images and target images on the DB often change to expect the accuracy of certain image processing. In the case of external installations that cannot be performed, the value of w is increased to increase the weight of the distance information, thereby improving the accuracy of the system.

As another embodiment, the positioning device 800 may recommend at least one location based on the final location of the target image.

After the positioning device 800 grasps the current location based on the GPS location information and the image information, a user may recommend a few nearby attractions that many people who took pictures at the current location have since gone to.

As another embodiment, the step in which the positioning device 800 recommends at least one location may be recommended using Equation 4 and Equation 5 below.

[Equation 4]

Order = freq * C ^{(distance * gain)} (however, 0 <gain <1)

Here, Order is the recommended score of the recommended place, freq is the frequency of visits to multiple users' recommended places, distance is the distance between the final location of the target image and the location of the recommended place, and gain is the weight.

[Equation 5]

및

은 대상이미지의 최종위치의 위도 및 경도,

및

는 추천된 장소의 위도 및 경도이다.here,

And

Is the latitude and longitude of the final position of the target image,

And

Is the latitude and longitude of the recommended place.

Through this, the user of the current location can be recommended at least one attraction based on the current location, and can generate an optimal tourism route.

8 is a block diagram illustrating a positioning device 800 according to an embodiment of the present invention.

Referring to FIG. 8, the positioning device 800 includes a receiving unit 810, an extraction unit 820, a selection unit 830, and a determination unit 840.

The receiving unit 810 receives a target image photographing a specific photographing location from an image photographing apparatus.

The extraction unit 820 extracts at least one feature point from the target image.

The sorting unit 830 selects at least one matching image among the plurality of images based on at least one feature point and feature points of a plurality of pre-stored images.

The determining unit 840 determines the final position of the target image by using the matching similarity between the target image and the image based on the feature information and the location information included in each of the at least one matching image.

In another embodiment, the positioning device 800 may further include a recommendation unit that recommends at least one location based on the final location of the target image.

As described above, in the specification and drawings, preferred embodiments of the present invention have been disclosed, but it is possible in the technical field to which the present invention pertains that other modified examples based on the technical spirit of the present invention may be implemented in the exceptions of the embodiments disclosed herein. It is obvious to those with ordinary knowledge. In addition, although specific terms are used in the present specification and drawings, they are merely used in a general sense to easily describe the technical contents of the present invention and help understanding of the present invention, and are not intended to limit the scope of the present invention.

800: positioning device
810: receiver
820: extraction unit
830: selector
840: Confirmation

Claims (8)

Receiving a target image photographing a specific photographing location from an image photographing apparatus;
Extracting at least one feature point from the target image;
Selecting at least one matching image among the plurality of images based on the at least one feature point and a feature point of a plurality of pre-stored images; And
And determining the final position of the target image using matching information between the target image and the location information included in each of the at least one matching image and the image based on the feature point. How to decide.
According to claim 1,
The feature point extraction is a location determination method using image processing and location information, characterized by using a scale-invariant feature transform (SIFT), speeded up robust features (SURF), or an Oriented FAST and Rotated BRIEF (ORB) feature point extraction algorithm.
According to claim 1,
The step of selecting the matching image,
Positioning method using image processing and location information, characterized by being performed by feature matching between comparison images.
According to claim 1,
The step of determining the final position of the target image by using the matching similarity between the images is Equation 1, Equation 2, and Equation 3, wherein the image processing and location determination method using location information.
[Equation 1]
I _Place (C) = MAX (I _V (P _i , C))
Here, I _Place (C) is the highest matching similarity, I _V (P _i , C) is the matching similarity between the at least one matching image and the target image, C is the target image, P _i is the i-th This is the matching image.
[Equation 2]
I _V (P _i , C) = wD (G (P _i ), G (C)) + (1-w) L _i (however 0 <w <1)
Here, w is a weight, D (G (P _i ), G (C)) is G (C) is location information of the target image, G (P _i ) is i-th matching image location information, D (G ( P _i ), G (C)) is the Euclidean distance between the location of the target image and the location of the matching image, and L _i is a softmax function for the number of optimal matching points of P _i .
[Equation 3]

Here, i is the i-th matching image, n _pl is the plurality of _pre -stored images, j is the j-th matching image among the pre-stored plurality of images, and f is the number of optimal matching points of the i or j-th matching image, respectively. .
According to claim 1,
Positioning method using image processing and location information, further comprising the step of recommending at least one location based on the final location of the target image.
The method of claim 5,
The step of recommending the at least one location is a method of positioning using image processing and location information, characterized in that it is recommended using Equation 4 and Equation 5 below.
[Equation 4]
Order = freq * C ^{(distance * gain)} (however, 0 <gain <1)
Here, Order is the recommended score of the recommended place, freq is the frequency of visits to the recommended places of multiple users, distance is the distance between the final location of the target image and the location of the recommended place, and gain is weight.
[Equation 5]

here,

And

Is the latitude and longitude of the final position of the target image,

And

Is the latitude and longitude of the recommended place.
A receiver configured to receive a target image photographing a specific photographing location from an image photographing apparatus;
An extraction unit for extracting at least one feature point from the target image;
A selector configured to select at least one matching image among the plurality of images based on the at least one feature point and a feature point of a plurality of pre-stored images; And
Position using image processing and location information including a determining unit that determines a final position of the target image using matching information between the target image and the location information included in each of the at least one matching image and the image based on the feature point Crystal device.
The method of claim 7,
And a recommendation unit for recommending at least one location based on the final location of the target image.

Images