KR20200012379A - Image-based indoor position detection apparatus and detection method - Google Patents

Abstract

The present invention relates to a device for detecting an indoor position based on an image using BIM information and a deep learning network and a method thereof. According to an embodiment of the present invention, the device for detecting an indoor position based on an image comprises: a server communication unit for receiving an image, photographed through a camera embedded in a user terminal, through wired/wireless communication; a server storage unit for storing a building information modeling database (BIM DB) in which indoor images within at least one building are stored; and a server control unit for extracting a layer including structure information from the images received from the user terminal, extracting an indoor image having the similarity equal to or greater than a reference value by matching the structure information on the extracted layer with the indoor images stored in the BIM DB, and detecting a current position of a user using the extracted indoor images.

Description

Image-based indoor position detection apparatus and detection method

The present invention relates to an image-based indoor position detection device and a detection method, and more particularly to an image-based indoor position detection device and a detection method using BIM information and deep learning network.

Finding the location and orientation of objects in a room is often used for augmented reality, indoor route guidance, and tracking of objects or people. Unlike outdoors where GNSS (global navigation satellite system) signals are mainly used for positioning, indoors cannot receive GNSS (global navigation satellite system) signals. Therefore, various studies for indoor positioning have been made.

Indoor positioning is mainly based on radio signals, but these methods require preliminary work in the room where positioning is required, and the costs are repeatedly generated by sensors that increase with area. In addition, since the position of the object is determined based on the signal, it is difficult to determine the direction of the object.

Image-based indoor positioning methods estimate the location of an object by comparing the indoor image at the time of location confirmation with previously constructed indoor image data. In addition to constructing image data, indoor pictures must be taken to create indoor maps. This also requires cumbersome work in advance for indoor positioning.

Building information modeling (BIM) refers to an integrated building modeling process from design to construction and maintenance of facilities after completion. The time and cost benefits of using BIM are demonstrated and are increasingly being used. Information management of integrated buildings is possible, and drawing extraction is easy at any point, which is useful for facility maintenance.

BIM can also be used for indoor positioning with sensors. Primarily, BIM is used to suggest structural information, visualize, and provide integrated information that can limit the sensor's margin of error. However, this also has the limitation of indoor positioning using a sensor.

Korea Patent Registration 10-1415016

An object of the present invention is to provide an image-based indoor position detection device and a detection method using a deep learning network and BIM information that can be implemented without using a sensor or a separate 3D modeling.

Image-based indoor position detection apparatus according to an embodiment of the present invention,

A server communication unit which receives an image captured by a camera embedded in a user terminal through wired or wireless communication; A server storage unit configured to store a building information modeling database (BIM DB) storing indoor images of at least one interior of the building; And extracting a layer including structure information from the image received from the user terminal, matching the extracted structure information with one or more indoor images stored in the BIM DB, and extracting an indoor image having a similarity or higher than a reference value. The server controller may include detecting a current location of the user using the extracted indoor image.

In the image-based indoor position detection apparatus according to an embodiment of the present invention, the BIM DB may store image layers obtained by performing a deep learning network on indoor images of the at least one building.

In the image-based indoor position detection apparatus according to an embodiment of the present invention, the deep learning network may be a convolutional neural network (CNN) including at least one of VGG-16 and VGG-19. have.

In the image-based indoor position detection apparatus according to an embodiment of the present invention, the BIM DB stores image layers obtained by performing a convolutional neural network for indoor images of the at least one building, and the server controller The layer extraction module for extracting a layer containing the structure information from the image received from the user terminal; An image matching module for extracting an image layer having a similarity or more than a reference value by matching the structure information of the extracted layer with the structure information of the image layers stored in the BIM DB; It may include a position detection module for detecting the current position of the user by using the extracted image layer.

In the image-based indoor position detection apparatus according to an embodiment of the present invention, the structural information may be a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.

In the image-based indoor position detection apparatus according to an embodiment of the present invention, the image layer may be a fourth pulling layer obtained by performing the convolutional neural network.

Image-based indoor position detection apparatus according to another embodiment of the present invention,

A communicator configured to receive BIM data for an indoor image of at least one building from an external server; Extract the structural information included in the image taken by the built-in camera, match the extracted structural information with any one or more indoor images included in the received BIM data to extract an indoor image having a similarity or more than a reference value, It includes a control unit for detecting the current location of the user using the indoor image.

In the image-based indoor position detection apparatus according to another embodiment of the present invention, the BIM data may be an image layer obtained by performing a deep learning network on the indoor images of the at least one building.

In the image-based indoor position detection apparatus according to another embodiment of the present invention, the deep learning network may be a convolutional neural network including at least one of VGG-16 and VGG-19.

In the image-based indoor position detection apparatus according to another embodiment of the present invention, the BIM data are image layers obtained by performing a convolutional neural network on the indoor images of the at least one building, the control unit, A layer extraction module for extracting structural information included in an image captured by the built-in camera; An image matching module which extracts an image layer having a similarity or higher than a reference value by matching the extracted structure information with structure information of image layers included in the BIM data; It may include a position detection module for detecting the current position of the user by using the extracted image layer.

In the image-based indoor position detection apparatus according to another embodiment of the present invention, the structural information may be a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.

In the image-based indoor position detection apparatus according to another embodiment of the present invention, the image layer may be a fourth pulling layer obtained by performing the convolutional neural network.

Image-based indoor location detection method according to an embodiment of the present invention,

Forming and storing a BIM DB in which indoor image layers inside at least one building are stored; Receiving an image photographed through a camera through wired or wireless communication; Extracting structural information from the received image; Extracting an image layer having a similarity or higher than a reference value by matching the extracted structure information with structure information of image layers stored in the BIM DB; Detecting a current location of a user using the extracted image layer.

In the image-based indoor location detection method according to an embodiment of the present invention, the BIM DB is a DB that stores the image layers obtained by performing a convolutional neural network for the indoor images of the at least one building, the The structural information may be a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.

In the image-based indoor position detection method according to an embodiment of the present invention, the image layer may be a fourth pulling layer obtained by performing the convolutional neural network.

Image-based indoor location detection method according to another embodiment of the present invention,

Receiving BIM data for at least one building from an external server; Extracting structural information included in an image captured by an embedded camera; Extracting an indoor image having a similarity or higher than a reference value by matching the extracted structural information with at least one indoor image included in the received BIM data; Detecting a current location of a user using the extracted image layer.

In the image-based indoor position detection method according to another embodiment of the present invention, the BIM data are image layers obtained by performing a convolutional neural network for the indoor images of the at least one building, the structure information is It may be a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.

In the image-based indoor position detection method according to another embodiment of the present invention, the image layer may be a fourth pulling layer obtained by performing the convolutional neural network.

One embodiment of the present invention, executed by a computer, forming and storing a BIM DB in which at least one indoor image layers inside a building are stored; Receiving an image photographed through a camera through wired or wireless communication; Extracting structural information from the received image; Extracting an image layer having a similarity or higher than a reference value by matching the extracted structure information with structure information of image layers stored in the BIM DB; It can be executed by a computer-readable recording medium recording a program for executing the step of detecting the current position of the user by using the extracted image layer.

Another embodiment of the present invention is implemented by a computer, receiving BIM data for at least one building from an external server; Extracting structural information included in an image captured by an embedded camera; Extracting an indoor image having a similarity or higher than a reference value by matching the extracted structural information with at least one indoor image included in the received BIM data; It can be executed by a computer-readable recording medium recording a program for executing the step of detecting the current position of the user by using the extracted image layer.

Other specific details of embodiments according to various aspects of the present invention are included in the following detailed description.

According to the image-based indoor position detection apparatus and detection method according to embodiments of the present invention,

Features obtained from a pre-learned deep learning network (specifically, CNN, more specifically, VGG network) can be used to identify the current location within the building through image extraction and matching.

Since each BIM image stored in the BIM DB is stored with indoor location and direction information, if the BIM image acquired in the same direction and the same direction as the indoor image captured by the camera is matched, the user's indoor image is captured. The location information can be confirmed accurately.

In addition, according to the image-based indoor position detection apparatus and detection method according to the embodiments of the present invention, since it utilizes the pre-made BIM modeling data, it is not necessary to produce a separate 3D modeling for the indoor position confirmation.

In addition, unlike the conventional vision-based localization method, it does not involve the trouble of constructing a DB by photographing various places in the room in advance.

1 is a block diagram showing an image-based indoor position detection system according to an embodiment of the present invention.
2 is a block diagram illustrating a server controller which is one component of an image-based indoor position detection apparatus according to an embodiment of the present invention.
3A is a diagram illustrating a process of constructing a BIM DB stored in a server storage, which is one component of an image-based indoor position detection apparatus according to an embodiment of the present invention.
3B is a diagram illustrating a multilayer structure of a convolutional neural network (CNN).
4 is a diagram illustrating the overall operation of the image-based indoor position detection apparatus according to an embodiment of the present invention.
5 is an exemplary diagram for describing a process of matching an indoor image stored in a BIM DB and an indoor image photographed by a user terminal in an image-based indoor position detection apparatus according to an embodiment of the present invention.
6 is a flowchart illustrating an image-based indoor position detection method according to an embodiment of the present invention.
7 is a control flowchart of an image-based indoor position detection method according to an embodiment of the present invention.
8 is a block diagram showing an image-based indoor position detection system according to another embodiment of the present invention.
9 is a block diagram showing an image-based indoor position detection apparatus according to another embodiment of the present invention.
10 is a control flowchart of an image-based indoor position detection method according to another embodiment of the present invention.

As the invention allows for various changes and numerous embodiments, particular embodiments will be illustrated and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present invention, the terms 'comprise' or 'having' are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof. Hereinafter, an image-based indoor position detection apparatus and a detection method according to an embodiment of the present invention will be described with reference to the drawings.

1 is a block diagram showing an image-based indoor position detection system according to an embodiment of the present invention, Figure 2 is a server control unit shown in one configuration of the image-based indoor position detection apparatus according to an embodiment of the present invention It is a block diagram.

As shown in FIG. 1, an image-based indoor location detection system according to an embodiment of the present invention includes a user terminal 100 and an indoor location detection server 200. Here, the image-based indoor position detection apparatus according to an embodiment of the present invention may be an indoor position detection server 200. On the other hand, the image-based indoor position detection apparatus according to another embodiment to be described later may be a user terminal 100, which will be described later. In this embodiment, the indoor location detection server 200 will be described as an image-based indoor location detection device.

The indoor location detection server 200 includes a server communication unit 210, a server storage unit 220, and a server control unit 230.

The server communication unit 210 is a communication means for performing a function of interworking with the user terminal 100 via a wired or wireless communication network, and performs a function of transmitting and receiving various data. The server communication unit 210 receives an image captured by a camera embedded in the user terminal 100, and transmits the current location information of the user detected by the server controller 230 to the user terminal 100.

The server storage unit 220 stores various data necessary for driving the indoor location detection server 200. In addition, the server storage unit 220 may store a program for performing a deep learning network. Here, the deep learning network may be a convolutional neural network (CNN). More specifically, the convolutional neural network may be at least one of VGG-16 and VGG-19. In the following description, the deep learning network will be described as an example of the VGG-16 network of the convolutional neural network, and the VGG-16 network is abbreviated as a VGG network. Of course, the deep learning network applicable in the present invention is not limited thereto.

3 is a diagram illustrating a process of constructing a BIM DB stored in a server storage unit, which is one component of an image-based indoor position detection apparatus according to an embodiment of the present invention.

The server storage unit 220 extracts an indoor image of the building from the BIM modeling data 221 for virtually modeling the building 222, and extracts the extracted image layer using the extracted indoor image as an input 223 of the VGG network. Fields 224a to 224e can be stored. The image layers 224a to 224e extracted through the VGG network V may be stored in the server storage unit 220 by BIM DB (Building Information Modeling Database). Each of the extracted image layers 224a to 224e includes an image feature map. In particular, the server storage unit 220 may store a fourth pooling layer 224d among the image layers extracted through the VGG network V.

On the other hand, it is possible to extract the image layers by driving the VGG network separately from the outside, the server storage unit 220 may be stored in the BIM DB extracted image layers.

That is, the server storage unit 220 stores image layers including an image feature map obtained as a result of passing through each layer of the convolutional neural network CNN as a BIM DB. Image feature maps are used to evaluate image similarity, which is vectorized and calculated as cosine distance.

The indoor image of the building included in the BIM modeling data has indoor location and orientation information, which is also included in the image layers 224a through 224e extracted through the VGG network V. Therefore, when the image most similar to the image photographed by the user terminal 100 in the interior of the building is detected in the BIM DB, the user's position can be detected therefrom.

The convolutional neural network (CNN) consists of a plurality of layers and learns to extract meaningful features of the image to perform the desired task. The types of layers constituting the CNN are largely divided into a convolutional layer, a pooling layer, and a fully-connected layer.

The CNN receives a two-dimensional raw image as an initial input, and each layer receives the output from the previous layer. In the convolutional layer, there is a filter having the size specified in each layer in the same number as the depth of the layer, and each filter is strides over the input. It is moved at intervals of and undergoes input and discrete convolution operations.

Since array data, such as images, have a higher correlation as the distance is closer, this operation extracts prominent features in the receptive field, which is the part of the filter that is applied at the input. .

Therefore, the output through the convolutional layer may be regarded as a feature extracted from each position of the input image. When the features of each part of the image are found in the convolutional layer, the pooling layer merges the semantically similar features into one. The network performance is invariant to image distortion and movement. In addition, since the dimension of the feature map is reduced, the area of the input image represented by the unit area of the feature map is larger than before.

When this feature map enters the input of the new convolutional layer, the feature is extracted from a larger area of the original image, even if the same size filter as the previous convolutional layer is applied. Therefore, when the convolutional layer and the pooling layer are repeated, features in a larger area are extracted from the original image, so that a global descriptor that observes the whole as the second half of the network can be obtained.

In the present invention, the current position of the user is estimated by using a feature map, which is a collection of image layers that pass through each layer of a pre-trained convolutional neural network.

CNNs trained for object classification show excellent results when applied to other types of image extraction. Since the CNN has learned a filter that can extract the main features of the input image during the learning process, the CNN can extract significant features in the image even when other types of images are input. Therefore, in the present invention, it is preferable to use a convolutional neural network among various deep learning networks in order to match a BIM image that is already constructed with an indoor image that is visually similar but there is a difference in image type.

3B shows the VGG-16 network structure, where several convolutional layers and max pooling are repeated five times and at the end of the network there are three fully-connected layers. do. The result of the pulley-connected layer is not used as a feature for position detection in the present invention because all of the features of the image are mixed to lose positional information, and the position detection is performed through five pooling layers that have been passed through max pooling. Used as a feature for. The numerals shown in FIG. 3 do not mean reference numerals, but are for explaining image size.

In the present invention, the data for image retrieval is composed of BIM images rendered similar to the user's field of view at a designated indoor location. BIM modeling image can be extracted inside the building, that is, indoor, pedestrian gaze in the form of a two-dimensional image. The direction of the pedestrian view to be extracted is variously set in advance and applied equally at each position. In addition, the BIM modeling image is constructed by rendering a user view in the BIM model according to the angle of view information of the camera to be used by the user, which is stored in the BIM DB.

Each BIM modeling image has rendered interior location and orientation information to present the user's current location information when matched with the picture taken by the user. In this case, a door or a wall of a single color that can be extracted in the same location may be removed from the BIM modeling image because it is not suitable for providing location information using image extraction.

The server controller 230 controls the overall functions of the indoor location detection server 200. The server controller 230 extracts a layer including structure information from an image received from the user terminal 100, and matches the extracted structure information with one or more image layers stored in a BIM DB to have a similarity or higher than a reference value. The image layer is extracted and the current position of the user is detected using the extracted image layer. Here, the structure information may mean the shape of the image feature map included in the layer. That is, a matching image layer is extracted by comparing the similarity of the overall shape of the image feature map.

To this end, the server controller 230 includes a layer extraction module 231, an image matching module 232, and a position detection module 233.

The layer extraction module 231 extracts a layer including structure information from an image received from the user terminal 100, and the image matching module 232 transmits the structure information of the extracted layer to any one or more image layers stored in a BIM DB. The image layer having a similarity or more than a reference value is extracted by matching the structure information included in the. In this case, the similarity may be calculated through cosine similarity evaluation using cosine distance between the vectorized images. Cosine distance is a cosine of the angle between two vectors, which can be used to evaluate similarity between vectors and has the advantage of being naturally normalized by the norm.

The position detection module 233 extracts position / direction information included in the extracted image layer to detect the current position of the user.

The function of the server controller 230 may be performed using a VGG network, which is a kind of a convolutional neural network. That is, the VGG network performs the functions of the layer extraction module 231 and the image matching module 232, and when the image layer is extracted as a result of driving the VGG network, the position detection module 233 performs the corresponding image layer. The location / direction information included in the extracted information is transmitted to the user terminal 100 through the server communication unit 210.

Here, the image layer having a similarity or more than a reference value may be the fourth pulling layer 224d among the image layers 224a to 224e extracted through the VGG network V. FIG. The inventor of the present invention has confirmed that, when the fourth pulling layer 224d is used, a matching result with high accuracy can be derived. This will be described later.

4 is a diagram illustrating the overall operation of the image-based indoor position detection apparatus according to an embodiment of the present invention.

When an image captured by a camera built in the user terminal 100 is received, the image is resized to a size suitable for the VGG network input (for example, a size of 224 × 224) and input to the VGG network (V).

The server controller 230 passes the input image through a plurality of convolutional layers and max pooling, and obtains a plurality of pooling layers 234a through 234e as a result. The feature for matching is obtained in the form of a feature map that is the result of passing through each layer except the fully-connected layer, located at the last end of the network.

The inventors of the present invention, based on the results of experiments for a number of pooling layers, among the image layers stored in the BIM DB obtained by pre-learning through the VGG network (V) "image layer having a similarity or more than the reference value" (that is, the position detection Valid image layers) were identified as the fourth pooling layers 224d and 234d. When image matching was performed by extracting features from the fourth pooling layer, an accurate result was obtained compared to other layers.

In this regard, a description will be given with reference to FIG. 5. FIG. 5 is an exemplary diagram for describing a process of matching an indoor image stored in a BIM DB and an indoor image photographed by a user terminal in an image-based indoor position detection apparatus according to an embodiment of the present invention.

FIG. 5 visualizes each image layer 224a through 224e and 234a through 234e in order to identify and compare features extracted from each image layer 224a through 224e and 234a through 234e. FIG. 5 is an example of visualizing characteristics obtained in each pooling layer when the indoor photograph and the BIM image acquired at the same point and direction pass through the VGG-16 network, respectively.

As the layer deepens, the CNN is trained to extract features from an increasingly wide area of the image. In other words, the network starts from extracting local descriptors and gradually extracts global descriptors.

Referring to the features visualized in each image layer in FIG. 5, it can be visually confirmed that the results obtained from the pooling layer in the second half show structural characteristics (structural information) rather than detailed characteristics in the image.

Looking at the result from the image layer of the first half, it can be seen that the color arrangement is different depending on the detailed factors such as the tile grid, nameplate, and roughness. However, in the present invention, since image matching is performed through image extraction for two different types of images, it is necessary to compare the overall structure of the images rather than the detailed features.

In fact, the experimental results confirmed that the feature derived from the fourth pooling layer extracting a relatively global descriptor in the VGG network is most suitable for cross-domain indoor image matching.

Referring to FIG. 5, it can be seen that the color arrangement of the visualized features of the two matching images is most similar in the result of the fourth pulling layer. Features obtained in the fifth pooling layer extract features of a wider area than those obtained in the fourth pooling layer. However, since the image matching and extraction accuracy is further inferior, the inventors of the present invention estimate that the fifth pulling layer has an adverse effect on excessively shortening the image information. Through this, it was confirmed that cross domain image matching is not always useful to use a global descriptor that represents a wider area unconditionally.

Next, an image-based indoor location detection method according to an embodiment of the present invention will be described with reference to FIGS. 6 and 7. 6 is a flowchart illustrating an image-based indoor location detection method according to an embodiment of the present invention, and FIG. 7 is a control flowchart of the image-based indoor location detection method according to an embodiment of the present invention.

6 and 7, first, a BIM DB is formed and stored (S110). That is, the indoor image of the building is extracted (222) from the BIM modeling data 221 for modeling the building virtually. The extracted image layers 224a to 224e are stored using the extracted indoor image as an input 223 of the VGG network. In this case, when the fourth pooling layer 224d is used among the extracted image layers 224a to 224e, a matching result with high accuracy can be derived, so that the BIM DB uses the fourth pooling layer extracted from each indoor image. 224d) only. The BIM DB is stored in the server storage unit 220 of the indoor location detection server 200.

Next, a query image is received from the user terminal 100. (S120) When a user wants to check the location of a building in a room, the indoor of the location is photographed by a camera built in the user terminal 100. The photographed image (query image) is transmitted to the indoor location detection server 200 through wired or wireless communication.

Next, the indoor location detection server 200 receiving the query image executes the VGG network using the query image as an input (S130), extracts the feature in the query image, and extracts the extracted feature from the feature of the indoor image stored in the BIM DB. Here, the feature for matching is the structure information in the query image and the structure information of the indoor image stored in the BIM DB. The structure information may mean the shape of the image feature map included in the layer. That is, the similarity between the overall shape of the image feature map is compared.

The server controller 230 passes the input query image through a plurality of convolutional layers and max pooling, and obtains a plurality of pooling layers 234a through 234e as a result. The VGG network pre-trained in the BIM DB formation process extracts a layer including structure information from a query image for matching, and matches the extracted layer structure information with an image layer of an indoor image stored in a BIM DB to match similarities above a reference value. Extract the image layer with. As a result of experiments on various pooling layers, it is preferable that an image layer having a similarity or higher than a reference value is a fourth pooling layer.

Next, when the indoor image similar to the query image is determined through matching, the current position of the user is detected by extracting the position / direction information included in the corresponding indoor image layer (S150). Send to 100.

Next, an image-based indoor position detection apparatus according to another embodiment of the present invention will be described with reference to FIGS. 8 and 9. 8 is a block diagram illustrating an image-based indoor position detection system according to another embodiment of the present invention, and FIG. 9 is a block diagram showing an image-based indoor position detection device according to another embodiment of the present invention.

As shown in FIG. 8, the image-based indoor location detection system according to another embodiment of the present invention includes a user terminal 100 and a BIM DB server 201. Here, the image-based indoor position detection apparatus according to another embodiment of the present invention may be a user terminal 100.

In the same manner as in the above-described embodiment, the BIM DB server 201 extracts an indoor image of the building from the BIM modeling data 221 for virtually modeling the building 222, and inputs the extracted indoor image to the VGG network. The image layers 224a to 224e extracted as 223 are converted into BIM DB and stored. The BIM DB server 201 may store image layers including an image feature map obtained as a result of passing through each layer of the convolutional neural network CNN.

However, unlike the above-described embodiment, the BIM DB server 201 receives the image captured by the user terminal 100 and does not detect the current location of the user, but the user terminal 100 needs to check the location. Receives BIM data for a specific building, and detects the current location of the user using an image layer obtained by performing a deep learning network in the user terminal 100.

That is, the above-described embodiment detects the current location of the user in the indoor location detection server 200, and the present embodiment detects the current location of the user in the user terminal 100.

To this end, the user terminal 100, as shown in FIG. 9, includes a communication unit 110, an input unit 120, an output unit 130, a storage unit 140, and a controller 150. do.

The communication unit 110 serves to support the transmission and reception of various information with the BIM DB server 201 through the wired or wireless communication network (N). The communication unit 110 requests BIM data transmission to the BIM DB server 201 and receives the BIM data transmitted from the BIM DB server 201.

The input unit 120 transmits a variety of information such as numeric and text information input from a user, a signal input in connection with various function settings, and function control of the user terminal 100 to the controller 150. The input unit 120 may include at least one of a key input means such as a keyboard or a keypad, a touch input means such as a touch sensor or a touch pad, a voice input means, a gyro sensor, a geomagnetic sensor, an acceleration sensor, a proximity sensor, and a camera. It may include a gesture input means comprising the above.

The output unit 130 outputs the input result of the input unit 120 and outputs the current position of the user detected by the control unit 150.

The storage 140 stores various data, programs, and applications necessary for driving the user terminal. In addition, the storage 140 may store a program for executing a deep learning network. Here, the deep learning network may be a convolutional neural network (CNN). More specifically, the convolutional neural network may be at least one of VGG-16 and VGG-19. The storage 140 stores BIM data transmitted from the BIM DB server 201.

The controller 150 extracts a layer including structure information from an indoor image captured by a camera embedded in the user terminal 100, and includes the structure information of the extracted layer and the BIM data transmitted from the BIM DB server 201. The image layer having a similarity or more than a reference value is matched with any one or more image layers, and the current position of the user is detected using the extracted image layer.

To this end, the controller 150 includes a layer extraction module 151, an image matching module 152, and a position detection module 153.

The layer extraction module 151 extracts a layer including structure information from an indoor image photographed by a camera, and the image matching module 152 transmits the structure information of the extracted layer to the BIM data transmitted from the BIM DB server 201. An image layer having a similarity or higher than a reference value is extracted by matching one or more included image layers. In this case, the similarity may be calculated through cosine similarity evaluation using cosine distance between the vectorized images.

The position detection module 153 extracts position / direction information included in the extracted image layer to detect a current position of the user.

The function of the controller 150 may be performed using a VGG network, which is a kind of a convolutional neural network. That is, the VGG network performs the functions of the layer extraction module 151 and the image matching module 152, and as a result of driving the VGG network, when any one image layer is extracted, the position detection module 153 performs the corresponding image layer. Extract the position / direction information included in the output through the output unit 130.

Next, an image-based indoor position detection method according to another embodiment of the present invention will be described with reference to FIG. 10. 10 is a control flowchart of an image-based indoor position detection method according to another embodiment of the present invention.

As shown in FIG. 10, first, a BIM DB is formed and stored in the BIM DB server 201. (S210) The BIM DB forming process is as described above.

Next, the BIM DB server 201 requests from the user terminal 100 to transmit the BIM data for a specific building that needs to be located. (S220)

If the user knows information for identifying a building, such as the name / address of the building, the user may request the BIM data transmission for a specific building by inputting and transmitting the corresponding information. If the user does not know the information for specifying the building, the BIM data transmission for the specific building may be requested through the GPS signal embedded in the user terminal 100. Since the GPS signal is not received in the building, the BIM data transmission may be requested by transmitting the last GPS signal reception information to the BIM DB server 201.

Next, the BIM DB server 201 transmits the BIM data of the building corresponding to the request of step S220 to the user terminal 100. (S230) At this time, the transmitted BIM data is each floor of the convolutional neural network (CNN). The image layers may include image feature maps obtained as a result of passing through the layer.

Next, after receiving the BIM data, the VGG network is executed by inputting an indoor image (query image) taken by a camera built in the user terminal 100 (S240), and the features in the indoor image (query image) The extracted feature is matched with the feature of the indoor image included in the BIM data received from the BIM DB server 201 (S250). Here, the feature for matching is the structure information and the BIM in the indoor image (query image). Structure information of the indoor image included in the data.

Next, when the indoor image similar to the query image is determined through matching, the current position of the user is detected by extracting the position / direction information included in the corresponding indoor image layer (S260). Output through 130.

According to the image-based indoor position detection apparatus and the detection method according to the embodiments of the present invention as described above, the feature obtained in the pre-learned deep learning network (specifically, CNN, more specifically VGG network) Image extraction and matching to determine the current location in the building. Since each BIM image stored in the BIM DB is stored together with indoor location and direction information, when a BIM image acquired in the same direction and the same direction as the indoor image captured by the camera is matched, the user's indoor image is captured. The location information can be confirmed accurately.

In addition, according to the image-based indoor position detection apparatus and detection method according to the embodiments of the present invention, since it utilizes the pre-made BIM modeling data, it is not necessary to produce a separate 3D modeling for the indoor position confirmation. In addition, unlike the conventional vision-based localization method, it does not involve the trouble of constructing a DB by photographing various places in the room in advance.

As mentioned above, although an embodiment of the present invention has been described, those of ordinary skill in the art may add, change, delete or add elements within the scope not departing from the spirit of the present invention described in the claims. The present invention may be modified and changed in various ways, etc., which will also be included within the scope of the present invention.

100: user terminal 110: communication unit
120: input unit 130: output unit
140: storage unit 150: control unit
151: layer extraction module 152: image matching module
153: position detection module
200: indoor location detection server 210: server communication unit
220: server storage unit 230: server control unit
231: layer extraction module 232: image matching module
233: position detection module
201: BIM DB Server

Claims (20)

A server communication unit which receives an image captured by a camera embedded in a user terminal through wired or wireless communication;
A server storage unit configured to store a building information modeling database (BIM DB) storing indoor images of at least one interior of the building; And,
Extracting a layer including structure information from the image received from the user terminal, and matching the structure information of the extracted layer with any one or more indoor images stored in the BIM DB to extract an indoor image having a similarity or more than a reference value. Server control unit to detect the current location of the user by using the indoor image
Image-based indoor position detection device comprising a.
The method according to claim 1, wherein the BIM DB,
And storing image layers obtained by performing a deep learning network on indoor images of the at least one building.
The method of claim 2, wherein the deep learning network,
An image-based indoor position detection device, which is a convolutional neural network (CNN) including at least one of VGG-16 and VGG-19.
The method according to claim 1,
The BIM DB stores image layers obtained by performing a convolutional neural network on indoor images of the at least one building,
The server control unit,
A layer extraction module for extracting a layer including structure information from the image received from the user terminal;
An image matching module for extracting an image layer having a similarity or more than a reference value by matching the structure information of the extracted layer with structure information of the image layers stored in the BIM DB;
Position detection module for detecting the current position of the user by using the extracted image layer
Image-based indoor position detection device comprising a.
The method according to claim 4, wherein the structure information,
An image-based indoor position detection device having a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.
The method of claim 5, wherein the image layer,
And a fourth pulling layer obtained by performing the convolutional neural network.
A communicator configured to receive BIM data for an indoor image of at least one building from an external server;
Extract the structural information included in the image taken by the built-in camera, extract the indoor image having a similarity or more than a reference value by matching the extracted structural information with any one or more indoor images included in the received BIM data, Control unit to detect the current location of the user by using the indoor image
Image-based indoor position detection device comprising a.
The method according to claim 7, wherein the BIM data,
And an image layer obtained by performing a deep learning network on the indoor images of the at least one building.
The method of claim 8, wherein the deep learning network,
An image-based indoor position detection device, which is a convolutional neural network (CNN) including at least one of VGG-16 and VGG-19.
The method according to claim 7,
The BIM data are image layers obtained by performing a convolutional neural network on indoor images inside the at least one building,
The control unit,
A layer extraction module for extracting structural information included in an image captured by the built-in camera;
An image matching module which extracts an image layer having a similarity or higher than a reference value by matching the extracted structure information with structure information of image layers included in the BIM data;
Position detection module for detecting the current position of the user by using the extracted image layer
Image-based indoor position detection device comprising a.
The method according to claim 10, wherein the structure information,
An image-based indoor position detection device having a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.
The method of claim 11, wherein the image layer,
And a fourth pulling layer obtained by performing the convolutional neural network.
Forming and storing a BIM DB in which indoor image layers inside at least one building are stored;
Receiving an image photographed through a camera through wired or wireless communication;
Extracting structural information from the received image;
Extracting an image layer having a similarity or higher than a reference value by matching the extracted structure information with structure information of image layers stored in the BIM DB;
Detecting a current position of a user using the extracted image layer
Image-based indoor location detection method comprising a.
The method according to claim 13,
The BIM DB is a DB storing image layers obtained by performing a convolutional neural network for indoor images of at least one building,
And the structure information is a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.
The method of claim 14, wherein the image layer,
And a fourth pulling layer obtained by performing the convolutional neural network.
Receiving BIM data for at least one building from an external server;
Extracting structural information included in an image captured by an embedded camera;
Extracting an indoor image having a similarity or higher than a reference value by matching the extracted structural information with at least one indoor image included in the received BIM data;
Detecting a current position of a user using the extracted image layer
Image-based indoor location detection method comprising a.
The method according to claim 16,
The BIM data are image layers obtained by performing a convolutional neural network on indoor images inside the at least one building,
And the structure information is a shape of an image feature map included in an image layer obtained by performing a convolutional neural network on the received image.
The method of claim 17, wherein the image layer,
And a fourth pulling layer obtained by performing the convolutional neural network.
Run by the computer,
Forming and storing a BIM DB in which indoor image layers inside at least one building are stored;
Receiving an image photographed through a camera through wired or wireless communication;
Extracting structural information from the received image;
Extracting an image layer having a similarity or higher than a reference value by matching the extracted structure information with structure information of image layers stored in the BIM DB;
Detecting a current position of a user using the extracted image layer
A computer readable recording medium having recorded thereon a program for executing the program.
Run by the computer,
Receiving BIM data for at least one building from an external server;
Extracting structural information included in an image captured by an embedded camera;
Extracting an indoor image having a similarity or higher than a reference value by matching the extracted structural information with at least one indoor image included in the received BIM data;
Detecting a current position of a user using the extracted image layer
A computer readable recording medium having recorded thereon a program for executing the program.

Images