KR20210022343A - Method and system for providing mixed reality contents related to underground facilities - Google Patents

Abstract

A method for providing mixed reality related to underground facilities according to an embodiment of the present invention is a method for providing mixed reality related to underground facilities managing and supervising facilities buried underground based on data and mixed reality technology related to underground facilities in a processor of a terminal. The method comprises the steps of: controlling a camera to obtain a captured image; detecting a reference point object by processing the captured image; recognizing the code displayed on a reference point in the reference point object; obtaining 3D spatial model information matching the recognized code; setting 3D spatial information for the captured image based on the obtained 3D spatial model information; and displaying a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the 3D spatial information.

Description

Method and system for providing mixed reality related to underground facilities {METHOD AND SYSTEM FOR PROVIDING MIXED REALITY CONTENTS RELATED TO UNDERGROUND FACILITIES}

The present invention relates to a method and system for providing mixed reality (MR) related to underground facilities. More specifically, it relates to a method and system for providing mixed reality related to underground facilities that manage and supervise facilities buried underground by fusion of data related to underground facilities and mixed reality technology in connection with a terminal.

In general, underground facilities, which are defined as facilities buried underground, such as water supply facilities, sewer facilities, gas facilities, communication facilities, and electric facilities, are essential infrastructure for operating cities, and unlike ground facilities, they are buried in the ground. Not only is it not easy to manage according to the variety of s, it is also difficult to derive a countermeasure for an accident.

In particular, in the event of an unexpected accident on an underground facility, it can be said that it is very important to systematically manage it as loss of life or property is caused.

Therefore, when underground facilities are buried, ground indicators are installed on the ground, or ground indicators are installed underground to provide information recognition means that can easily detect the movement of the location of underground facilities in the event of natural disasters such as earthquakes, landslides, and floods. Various management systems have been proposed.

In a conventional management system, a location corresponding to an underground facility is recognized through a global positioning system (GPS), or an information recognition means such as a barcode or QR code is installed on the ground and an underground facility matched with the information recognition means. By constructing a database with related data, the location of underground facilities is recognized.

Thereafter, the field worker or manager obtains the location information and attribute information through the above-described GPS location information, tag recognition for the information recognition means, or wireless communication method tag recognition using the terminal, and searches the database using this. By doing so, it is possible to perform management work on underground facilities by receiving information on stored underground facilities.

However, according to the currently provided underground facility information, it remains at the level of displaying only its location and approximate shape by simply mapping the corresponding underground facility on the map, so what three-dimensional form the facilities in the underground space are arranged in? There is a limitation that it is difficult to easily identify.

In addition, when mapping underground facilities based on the location recognized through the satellite navigation system where errors inevitably occur, it is difficult to perform the management work on the underground facilities because the exact location of the underground facilities cannot be displayed due to the error. There is a problem, and there is a problem that it is difficult to use in an urban area where the satellite navigation system does not work well.

In addition, it is difficult to separately install information recognition means such as barcodes or QR codes on the ground, and if the information recognition means exposed on the ground is damaged, there is a problem that it becomes useless.

In addition, the conventional management system maps and provides pre-stored underground facility information on a map, but interacts with information related to underground facilities in consideration of real-time changes in the location and/or orientation (position) of the user terminal using the underground facility information. It is insufficient to provide (interaction).

In particular, in the absence of GPS, in the absence of GPS, it is difficult to accurately provide interactive underground facility information according to changes in the user's location and/or the camera orientation (position) of the terminal, so it is necessary to introduce a technology to solve this problem. .

In addition, in the electronic map provided by private companies, the location of objects such as terrain and buildings appearing on the map is set in relative coordinates, so the distance between underground facilities and the azimuth angles of each other do not match at the corresponding coordinates. Cases often arise.

In addition, the underground facility information stored in the database of the conventional underground facility management system is requested to a private company, or the location information, topographic information, and attribute information of the underground facilities are mapped on a 2D or 3D electronic map built by the company itself. As a result, it has a limitation in that its accuracy is remarkably low.

On the other hand, recently, when shooting using a camera module, content provision using a mixed reality (MR) technique that provides an environment in which a virtual world can be grafted to the real world to interact with a physical object and a virtual object has been actively provided. It is being studied.

In detail, mixed reality (MR) includes the meaning of augmented reality (AR: Augmented Reality) that adds virtual information based on reality and augmented virtuality (AV: Augmented Reality) that adds reality information to a virtual environment. It provides a smart environment in which virtual and virtual connections are naturally connected, allowing users to have a rich experience.

10-2018-0132183 A

The present invention, as conceived to solve the above problems, integrates data related to underground facilities and mixed reality (MR) technology in connection with a user's terminal to easily manage and supervise underground facilities related to underground facilities. Its purpose is to provide a method and system for providing mixed reality.

In detail, an object of the present invention is to provide information on underground facilities according to changes in the location and/or orientation (position) of the user terminal by using a mixed reality by interacting with the user's terminal in real time.

In addition, an embodiment of the present invention is to provide a method and system for providing mixed reality related to underground facilities that implement image recognition, tracking, and learning technology and 3D space recognition and camera tracking technology for providing information on underground facilities. have.

In addition, an embodiment of the present invention is a mixed reality related to underground facilities that easily provides information on underground facilities according to a change in the location and/or orientation (position) of a user terminal even in the absence of a satellite navigation system (GPS) in a mixed reality. Its purpose is to provide a provision method and system.

However, the technical problems to be achieved by the present invention and the embodiments of the present invention are not limited to the technical problems as described above, and other technical problems may exist.

The method for providing mixed reality related to underground facilities according to an embodiment is a method of providing a mixed reality related to underground facilities in which a processor of a terminal manages and supervises facilities buried in the basement based on data related to underground facilities and mixed reality technology, and includes a camera. Controlling to obtain a photographed image; Detecting a reference point object by image processing the captured image; Recognizing a code displayed at a reference point in the reference point object; Obtaining 3D spatial model information matching the recognized code; Setting 3D spatial information for the captured image based on the acquired 3D spatial model information; And displaying a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the 3D spatial information.

In this case, acquiring 3D spatial model information matching the recognized code may include acquiring 3D coordinates of the reference point object specifying a 3D space of the captured image.

In addition, setting 3D spatial information for the captured image may include setting a 3D spatial coordinate system for the captured image based on the 3D coordinates of the acquired reference point object.

In addition, obtaining the 3D spatial model information matching the recognized code further includes acquiring a scale of the reference point object, and setting 3D spatial information for the captured image May include obtaining the reference point scale information based on the size of the reference point object and a pixel displaying the reference point object.

In addition, the displaying of a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the 3D spatial information may include determining the size of the virtual object matching the 3D space of the captured image. It may include augmenting the captured image according to the reference point scale information.

In addition, the steps of updating the 3D spatial information by tracking changes in the position and orientation of the camera, and rendering and displaying a virtual object updated according to the updated 3D spatial information on the captured image It may contain more.

In addition, the updating of the 3D spatial information by tracking changes in the position and orientation of the camera may include updating the 3D spatial information based on real-time sensing data of an inertial sensor (IMU), and the The updated 3D spatial information may include correcting the updated 3D spatial information based on the position and direction of the reference point object recognized based on a Visual Inertial Odometry (VIO) technology.

The method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention allows for easy management and supervision of underground facilities by fusion of data related to underground facilities and mixed reality (MR) technology in connection with a user's terminal. By doing so, real-time information on underground facilities such as water and sewage, electricity, communication, gas pipes, building electrical wiring, heating and cooling pipes, etc. can be provided to users by 3D visualization based on mixed reality technology. There is an effect of building a mixed reality service infrastructure that provides proof contents specialized for inspection and maintenance.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention, by interacting with the user's terminal in real time, are updated in real time according to the change in the location and/or orientation (position) of the user terminal. By providing the information using mixed reality, there is an effect that the user can accurately and conveniently provide information on the underground facilities matching the space in which the user wants to check the information on the underground facilities.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention have improved accuracy by matching and matching 3D space and underground facilities using DGPS (Differential Global Positioning System, high-precision GPS). It has the effect of providing information on underground facilities.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention implements image recognition/tracking/learning technology and 3D space recognition/camera tracking technology for providing information on underground facilities, thereby providing high-quality underground facilities. Facility information can be provided, and through this, there is an effect of maximizing the efficiency of underground facility investigation, analysis, design, education prevention, and supervision work.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention can use a camera and/or an inertial sensor (IMU) of the terminal to determine the location and/or orientation (posture) of the user terminal. By providing information on underground facilities according to changes, it is possible to provide information on underground facilities according to changes in the state of the user terminal through its own algorithm and process even without a separate location tracking device.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention provides an underground facility buried underground based on 3D spatial information provided by the national spatial information platform (V-WORLD) on an electronic map drawing. By mapping, it is possible to provide a more realistic and accurate shape of underground facilities including digital terrain, and there is an effect of ensuring compatibility and performance of existing infrastructure and new software (S/W).

However, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood from the following description.

1 is a conceptual diagram of a method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention.
2 is an internal block diagram of a terminal according to an embodiment of the present invention.
3 is a block diagram of an underground facility content providing server according to an embodiment of the present invention.
4 is a flowchart illustrating a method of constructing an underground facility database according to an embodiment of the present invention.
5 is a conceptual diagram illustrating a method of providing mixed reality related to underground facilities based on non-GPS connection according to an embodiment of the present invention.
6 is a flowchart illustrating a method of providing mixed reality related to underground facilities based on non-GPS connection according to an embodiment of the present invention.
7 is an example of a reference point according to an embodiment of the present invention, and is a view for explaining a method of automatically extracting an effective area from the reference point.
8 is an exemplary diagram for selecting an effective elliptical region through an Intersection Of Union (IOU) according to an embodiment of the present invention.
9 is a view for explaining a process of performing a homography conversion in order to convert an elliptical reference point into a circular reference point according to an embodiment of the present invention.
10 is an example showing 3D spatial model information of a reference point according to an embodiment of the present invention.
11 is an example of displaying a rendered virtual object on a captured image according to an embodiment of the present invention.
12 is a flowchart illustrating a method of providing mixed reality related to underground facilities based on GPS linkage according to another embodiment of the present invention.
13 is an example of acquiring 3D spatial information based on high-precision GPS (DGPS) according to another embodiment of the present invention.
14 is an example of tracking changes in position and orientation according to movement of a terminal based on a plurality of reference points based on a Visual Inertial Odometry (VIO) technology according to another embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments are illustrated in the drawings and will be described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them will become apparent with reference to the embodiments described below in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used for the purpose of distinguishing one constituent element from other constituent elements rather than a limiting meaning. In addition, expressions in the singular include plural expressions unless the context clearly indicates otherwise. In addition, terms such as include or have means that the features or components described in the specification are present, and do not preclude the possibility of adding one or more other features or components in advance. In addition, in the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, and thus the present invention is not necessarily limited to what is shown.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding constituent elements are assigned the same reference numerals, and redundant descriptions thereof will be omitted. .

1 is a conceptual diagram of a method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention.

Referring to FIG. 1, a method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention may include a terminal 100 and a content providing server 200 for underground facilities (hereinafter, referred to as a content providing server).

Here, each component of FIG. 1 may be connected through a network. The network refers to a connection structure in which information exchange is possible between respective nodes such as the terminal 100 and the content providing server 200, and examples of such networks include a 3GPP (3rd Generation Partnership Project) network, and LTE (Long Term). Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth A (Bluetooth) network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, and the like are included, but are not limited thereto.

-Terminal

First, in an embodiment of the present invention, the terminal 100 is a portable computing device possessed by a user who performs work at a site where underground facilities in each region are buried, and a program for performing a mixed reality service related to underground facilities is provided. Installed portable terminals such as smart phones, digital broadcasting terminals, mobile phones, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation, tablet PCs, wearable devices and smart glass, etc. It may include. Here, the underground facility may mean a facility buried underground for water supply, sewage, electricity, communication, gas, oil supply, and heating.

The terminal 100 may provide information on underground facilities including various types of information related to underground facilities based on a code matched with a reference point 10 included in an image photographing a specific space in an embodiment of the present invention. .

For example, the reference point 10 may include at least one of a mark nail, a buried pin, a mark pole, a mark pin, a manhole, and a mark nail exposed to the ground while being linked to an underground facility, and such a reference point 10 The role indicated by the reference point 10 may be marked with a first code, and for example, a first code indicating a water supply, a sewer, a sewage pipe, a street light line, an underground transmission line, or a communication line is engraved on a sign nail made of metal. Or it can be marked with relief.

In addition, at the reference point 10, a second code indicating the area where the reference point 10 is installed may be displayed. For example, a second code indicating the area name, address name, or road address name is engraved on a sign made of metal. Or it can be marked with relief.

In addition, a third code indicating the pipe diameter or pipe direction of the pipe connected to the reference point 10 may be displayed at the reference point 10. For example, a pipe diameter value or an arrow indicating the pipe direction may be engraved or It can be marked with relief.

This reference point 10 serves as a reference indicator for recognizing a three-dimensional space of a captured image captured through a camera, and information on the reference point 10 and the underground facility matching the reference point 10 is obtained. It may be a mark including the above possible codes.

In addition, here, the underground facility information is 3D modeling image (virtual object) of the underground facility, absolute location information, type information, size information, orientation information, underground analysis information (type of medium, depth of medium, etc.), completion information ( It may be information including at least one of a construction drawing, a construction location, a construction date, etc.), management information (management history, etc.), and/or information on an electronic drawing of an underground facility (a set of virtual objects).

The terminal 100 may provide information on such underground facilities by 3D visualization based on mixed reality (MR) technology. In an embodiment, the terminal 100 may output and provide information on underground facilities as a 3D image on a captured image captured by a camera based on a mixed reality technology.

In detail, in the terminal 100 according to an embodiment of the present invention, in providing information on underground facilities as a 3D image, a background image being photographed at a current location by a camera mounted on the terminal 100 (that is, the Set) can be provided as a mixed reality image by superimposing information on underground facilities (for example, a virtual object implemented as a 3D modeling image).

In this case, the virtual object according to the embodiment may mean a virtual object that is not visible on the 3D space of the background image, but is matched to the corresponding 3D space and stored in the database. For example, the virtual object may be a 3D modeling image of an invisible underground facility arranged underground in a corresponding 3D space. Here, the 3D modeling image may be an image created by a mathematical model capable of reproducing the shape of an underground facility in a virtual 3D space. That is, the 3D modeling image represents the shape of the underground facility. It may be data in a form that can be understood by a computer generated to be checked in a dimensional space.

In addition, the real object according to the embodiment may mean an object that is actually visible in a three-dimensional space of a background image, for example, a reference point including a sign nail, a manhole, etc. existing in the corresponding three-dimensional space. (10) and the like.

In addition, the terminal 100 may display information related to an underground facility as a mixed reality image in which a real object, which is a real physical object, and a virtual object, which is a virtual object generated based on information on the underground facility, can interact.

For example, when the underground facility is a pipe, the terminal 100 may display information on a pipe provided in the underground space of the current location in the form of a 3D pipe network or an augmented reality view. That is, the user may be provided with information on underground facilities buried based on his/her location through the terminal 100, and may be provided with a 3D modeling image whose viewpoint changes according to the photographing direction of the camera.

For example, when the terminal 100 photographs the ground, information on the underground facilities of the underground facilities matching the ground currently illuminated by the user may be displayed through the display in consideration of the current location of the terminal 100. In addition, when the user selects a specific coordinate of the electronic map displayed on the terminal 100, the terminal 100 displays a virtual underground space based on the coordinates and information on the underground facilities located within the corresponding underground space. Can also be displayed.

In addition, the terminal 100 may acquire identification information by recognizing a code (mark) for an underground facility in the field and transmit it to the content providing server 200 through an information communication network, and a three-dimensional matched to the identification information Spatial model information can be received and utilized.

Here, the 3D spatial model information is information matched with each reference point 10 and stored in a memory and/or a database, and a 3D coordinate (ie, absolute position) of the reference point 10 specifying a 3D space, and a reference point It may include information on underground facilities matching the scale of (10) and/or the reference point (10).

The terminal 100 according to an embodiment may acquire identification information by photographing a code and transmit it to a content management server. At this time, the content management server may read 3D spatial model information matching the received identification information from the database and provide it to the terminal 100, through which the terminal 100 obtains information on underground facilities. You can make use of it.

Here, in order to implement the above-described function, the terminal 100 may interwork with a content management server through a network and mount a recording medium in which an application program for displaying received information and reproducing a 3D modeled image is recorded.

In the above, the terminal 100 acquires identification information by recognizing the code of the reference point 10 matched with the underground facility, and transmits the obtained identification information to the content providing server 200 to receive 3D spatial model information. Although it has been described that information on underground facilities is acquired based on this, various embodiments may also be possible, such as being able to perform a series of operations of the content providing server 200 for obtaining information on underground facilities by itself. .

In addition, in the embodiment, the terminal 100 may obtain information on underground facilities updated in real time according to a change in the location and/or orientation (position) of the terminal 100, and based on the obtained information, the user's terminal (100) Underground facility information optimized for the state (ie, the location and/or orientation of the terminal 100) can be provided to the user through mixed reality.

As described above, according to the above-described structure, the terminal 100 according to an embodiment of the present invention provides information on underground facilities matching a specific space to the user as a three-dimensional image according to the state of the user's terminal 100, It is possible to easily check the buried condition of underground facilities.

Hereinafter, the structure of the terminal 100 according to an embodiment of the present invention will be described in detail with reference to the drawings. 2 is an internal block diagram of a terminal 100 according to an embodiment of the present invention.

Referring to FIG. 2, in an embodiment of the present invention, the terminal 100 includes a communication unit 110, a display unit 120, an interface unit 130, an inertial measurement unit (IMU), and a camera 150. ), a memory 160 and a processor 170.

First, the communication unit 110 may transmit and receive various data and/or information for providing a mixed reality service related to underground facilities.

In an embodiment, the communication unit 110 may connect to an information communication network and communicate with the content providing server 200 to transmit and receive data (eg, information on underground facilities) related to an underground facility-related mixed reality providing service.

These communication units 110, technical standards or communication methods for mobile communication (for example, GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), HSDPA (High Speed Downlink Packet Access), HSUPA ( High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.) Can transmit and receive signals.

Next, the display unit 120 may output various information related to a mixed reality providing service related to underground facilities as a graphic image.

In an embodiment, the display unit 120 may display a mixed reality image configured by the processor 170 of the terminal 100 including a graphic user interface (GUI).

The display unit 120 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. It may include at least one of a flexible display, a 3D display, and an e-ink display.

Next, the interface unit 130 may receive an input of a user's manipulation related to a mixed reality providing service related to an underground facility.

The interface unit 130 is a touch screen that is configured integrally with the display unit 120 as well as a key button provided in the terminal 100 to click a virtual button displayed on the screen. It can be implemented in the form of.

Next, the inertial sensor (IMU, 140) may be implemented including an acceleration sensor and/or a gyro sensor, etc., and may measure a change in the position and/or orientation of the terminal 100.

In an embodiment, the inertial sensor (IMU, 140) transmits the current azimuth data to the processor 170 when the terminal 100 displays the mixed reality image, so that a 3D modeling image (virtual object) corresponding to the time point is displayed. Can be output.

Next, the camera 150 may acquire an image by photographing a direction side disposed on the front side and/or the rear side of the terminal 100.

In an embodiment, the camera 150 may photograph a space (area) in which the user wants to obtain information on underground facilities and/or a code of the reference point 10 matched with the underground facility.

In particular, when displaying a mixed reality image based on a photographed image of a surrounding space of an area in which the terminal 100 is located, the camera 150 according to the embodiment may allow information on underground facilities to be superimposed on the photographed image to be displayed. have.

Next, the memory 160 may store any one or more of an operating system (OS), various application programs, data, and commands for providing a mixed reality providing service related to underground facilities.

The memory 160 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and is a web storage that performs a storage function of the memory 160 on the Internet. May be. In addition, the memory 160 may be a recording medium in a detachable form in the terminal 100.

Next, the processor 170 may control the overall operation of each of the above-described components in order to implement a mixed reality providing service related to underground facilities.

That is, the processor 170 may be a system-on-chip (SOC) suitable for the portable terminal 100 including a central processing unit (CPU) and a graphic processing unit (GPU), and the operating system stored in the memory 160 ( OS) and application programs, and the like, and control each component mounted on the terminal 100.

The processor 170 may communicate with each component internally through a system bus, and may include one or more predetermined bus structures including a local bus.

In addition, these processors 170, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers (controllers) ), micro-controllers, microprocessors, and electrical units for performing other functions.

-Server for providing contents of underground facilities

Meanwhile, in an embodiment of the present invention, the content providing server 200 may provide various types of information generated on an underground facility database based on spatial information related to a plurality of underground facilities.

In detail, the content providing server 200 stores and manages various information related to underground facilities in a database based on spatial information related to underground facilities obtained from the national spatial information platform (V-WORLD) and/or the terminal 100 It can be done, and the stored information can be transmitted to the terminal 100 and provided to the user. A detailed description of this will be described later.

Here, the above-described spatial information related to underground facilities may be a computer file provided by a national spatial information portal or a department operated by the Ministry of Land, Infrastructure and Transport, or a computer file based on an image acquired through the camera 150 of the terminal 100. May be.

3 is an internal block diagram of an underground facility content providing server 200 according to an embodiment of the present invention. In detail, referring to FIG. 3, in an embodiment of the present invention, the content providing server 200 may include a data receiving unit 210, a data transmitting unit 220, a data processing unit 230, and a database 240.

First, the data receiving unit 210 and the data transmitting unit 220 may exchange various data for performing a mixed reality service related to underground facilities with the terminal 100 and/or an external server through a network.

Next, the data processing unit 230 may perform a series of data processing for performing a mixed reality provision service related to underground facilities.

In this case, the data processing unit 230 may include a DB (database) construction unit 231, a three-dimensional space sensing unit 232, and an information arranging unit 233.

In detail, first, the DB construction unit 231 may generate information for constructing an underground facility database based on the information on the underground facility, and may convert the generated information into a database and store it in the database 240.

In detail, the DB construction unit 231 generates an information input unit for receiving spatial information related to underground facilities transmitted through an information communication network or stored in a recording medium, and a 3D modeling object (virtual object) corresponding to a plurality of underground facilities. A 3D rendering unit, a spatial information mapping unit that maps a 3D modeling object in a 3D virtual space defined by spatial information, and a reference point 10 providing unit that manages information related to the reference point 10 matching underground facilities Can include.

At this time, the above-described reference point 10 providing unit may include three-dimensional coordinates (absolute position) for each reference point 10, a scale of the reference point 10, and information on underground facilities matching the reference point 10. And, based on this, 3D spatial model information can be generated and provided.

In addition, the 3D space detection unit 232 may determine a 3D space matching the corresponding captured image based on the captured image acquired from the terminal 100 and spatial information related to underground facilities previously stored in the database 240. have.

In addition, the 3D space detector 232 may read a virtual object matching the 3D space from the database 240 and generate scene specification information corresponding to the 3D space based on this.

Here, the scene specification information is information that detects a virtual object matching the 3D space photographed by the camera 150 of the terminal 100 and assists in matching the detected virtual object with the corresponding 3D space. A detailed description will be provided later.

In addition, the information arrangement unit 233 renders each virtual object to be placed at an appropriate position in the corresponding 3D space based on the scene specification information and underground facility information generated through the 3D space detection unit 232 ( rendering).

In addition, the data processing unit 230 including the configuration unit as described above, in the embodiment of the present invention, according to the real-time position and / or orientation change according to the movement of the terminal 100, the scene specification information and the rendered virtual object. It can be updated in real time.

The data processing unit 230 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. , Micro-controllers, microprocessors, and may be implemented using at least one of electrical units for performing other functions.

Finally, the database 240 may store various data related to a mixed reality provision service related to underground facilities.

In an embodiment, in the database 240, an underground facility database including virtual objects related to a plurality of underground facilities, 3D spatial model information, and/or underground facility information may be constructed, stored, and managed.

The database 240 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and is a web storage that performs a storage function of the database 240 on the Internet. May be.

In more detail, in an embodiment of the present invention, the database 240 may match and store the 3D coordinates of the reference point 10 included in the 3D space with a corresponding 3D space.

In addition, the database 240 may match and store the absolute coordinates of a virtual object (ie, a three-dimensional modeling image of an underground facility) matched with the three-dimensional space.

In addition, the database 240 is an electronic drawing, which is a map generated by mapping the three-dimensional space and the underground facilities buried in the basement based on the spatial information related to the underground facilities provided by the national spatial information platform (V-WORLD). Can be saved.

Here, the electronic drawing may be information obtained by implementing a three-dimensional graphic map of an underground facility buried in the basement, and may be a set of virtual objects in an embodiment.

In addition, the database 240 may include an underground facility database including a plurality of pieces of information as described above.

Hereinafter, an embodiment of constructing an underground facility database in the database 240 will be described in detail with reference to the accompanying drawings.

4 is a flowchart illustrating a method of constructing an underground facility database according to an embodiment of the present invention.

Referring to FIG. 4, first, the content providing server 200 may acquire underground facility data based on a network map of underground facilities held by a local government (hereinafter, referred to as a local government). (S101)

Here, the underground facility data is information included in the underground facility information, and may include absolute location, type, size, and/or orientation information of the underground facility.

In detail, the content providing server 200 communicates with an underground facility-related server operated by a local government and/or a national geospatial information platform server to obtain underground facility data including an underground facility pipe network, etc. And, based on this, information on underground facilities can be constructed.

For example, the content providing server 200 may receive a water and sewage pipe network map from an underground facility-related server operated by a local government to construct an underground facility electronic drawing.

In addition, the content providing server 200 may acquire underground facility data from a management entity of the underground facility. (S103)

In detail, the content providing server 200 communicates with a server of a specialized institution that manages underground facilities such as Korea Electric Power Corporation and Korea Gas Corporation to obtain drawings or documents related to underground facilities, and based on this, information on underground facilities Can build.

In addition, the content providing server 200 may acquire underground facility data by measuring the current status of codes for each underground facility disposed on the site. (S105)

In detail, the content providing server 200 may generate information on underground facilities by identifying the status of codes assigned to the reference point 10 that is matched to each of the underground facilities and installed on the site.

For example, the content providing server 200, based on identification information and/or 3D spatial model information stored on the database 240 in response to the code assigned to the reference point 10 matching each underground facility, It is possible to obtain information such as the number of beacons and/or manholes installed on the site, the absolute location and/or type of underground facilities matched to the beacons and/or manholes, and build information on underground facilities based on this. have.

That is, the content providing server 200 may build underground facility information based on the underground facility data obtained in various ways as described above, and store the constructed underground facility information in the database 240. (S107)

In detail, the content providing server 200 can generate a 3D modeling image (virtual object) for each underground facility, 3D spatial model information, and/or information on underground facilities, based on the acquired underground facility data. As a result, a database for underground facilities can be built.

In addition, the content providing server 200 may additionally collect underground facility data by performing ground penetrating radar (GPR) and store it in the database 240. (S109)

In detail, the method and system for providing mixed reality related to underground facilities may further include an underground facility surveying and exploration device according to an embodiment.

Here, the underground facility surveying and exploration apparatus may acquire underground facility data by measuring the underground facility using the cadastral reference point 10 or exploring the underground facility through electronic induction. In addition, the underground facility surveying and exploration apparatus may transmit the acquired underground facility data to the content providing server 200 through a network.

In more detail, the content providing server 200 may receive the underground facility data obtained from the underground facility surveying and exploration apparatus.

At this time, in the embodiment, the underground facility surveying and exploration apparatus may detect a code given to the reference point 10 that exists in the three-dimensional space, and a three-dimensional space matched with the reference point 10 based on the detected code. Model information and the like can be obtained.

In addition, in an embodiment, the underground facility surveying apparatus may acquire information on the absolute location, orientation, and/or size of the underground facility buried in the basement based on the electromagnetic guided survey. In addition, in an embodiment, the apparatus for surveying and exploring underground facilities may acquire absolute location and/or orientation information of an underground facility with respect to a space in which the survey is performed in connection with a GPS.

Thereafter, the content providing server 200 may store the additionally collected underground facility data in the database 240, and update the underground facility information based on the additionally collected underground facility data.

In addition, the content providing server 200 may build an underground facility database based on the acquired information on the underground facility. (S111)

In detail, the content providing server 200 may build an underground facility database including a virtual object category, a 3D spatial model information category, and/or an underground facility information category.

In more detail, the underground facility database may store a virtual object in which an underground facility is modeled in three dimensions in a virtual object category.

In addition, the underground facility database includes the three-dimensional coordinates (ie, absolute position) of the reference point 10 that specifies the three-dimensional space matched with the reference point 10 in the 3D spatial model information category, and the size of the reference point 10 ( scale) and/or information on underground facilities matching the reference point 10 may be stored.

In addition, the underground facility database is a three-dimensional modeling image (virtual object) of the underground facility in the underground facility information category, absolute location information, type information, size information, orientation information, underground analysis information (type of medium, depth of medium, etc.). ), construction information (construction drawings, construction locations, completion dates, etc.), management information (management history, etc.), and/or information on underground facilities including at least one or more of information on electronic drawings of underground facilities (set of virtual objects). .

In this way, the content providing server 200 can systematically manage the data that is the basis of the underground facility related mixed reality providing service by constructing the underground facility database for the underground facility related mixed reality providing service as described above. And, by providing high-quality data, the quality of the mixed reality service related to underground facilities can be improved.

In the above, the content providing server 200 constructs an underground facility database, detects a three-dimensional space to generate scene specification information, and renders a virtual object based on the generated scene specification information and underground facility information, etc. Although it has been described that a series of operations are performed, various embodiments may also be possible, such as being able to perform some of the operations performed by the content providing server 200 in the terminal 100.

-Method of providing mixed reality related to underground facilities based on non -GPS connection

Hereinafter, a method for providing mixed reality related to underground facilities based on non-interworking with a satellite navigation system (GPS) will be described in detail with reference to the accompanying drawings. In more detail, a method of acquiring the initial orientation (position) between the space-camera 150 by recognizing the reference point 10 without interlocking with the GPS, and providing a mixed reality related to an underground facility based on this will be described in detail.

5 is a conceptual diagram illustrating a method of providing a mixed reality related to underground facilities based on non-GPS linkage according to an embodiment of the present invention, and FIG. 6 is a conceptual diagram illustrating a mixed reality related to underground facilities based on a non-GPS link according to an embodiment of the present invention. This is a flow chart for explaining the method.

5 and 6, first, the terminal 100 may acquire a captured image by controlling the camera 150. (S201)

In detail, the terminal 100 may acquire a photographed image captured by the camera 150 of a space (area) in which the user wants to obtain information on underground facilities.

In addition, the terminal 100 may image-process the acquired captured image to detect an object of the reference point 10. (S203)

In detail, the terminal 100 first detects an object to be identified within the captured image, identifies the real object, matches each underground facility among the identified real objects, and is installed on the site, thereby specifying a reference point 10 for specifying each underground facility. ) Objects can be detected.

Here, the reference point 10 may be a reference indicator on the ground for recognizing a three-dimensional space of the captured image, and at least the reference point 10 and information on the underground facilities matching the reference point 10 can be obtained. It may contain one or more codes.

For example, referring to FIG. 7, the reference point 10 may be a sign nail installed on the site, and may be implemented by including at least one or more of a plurality of codes (c1, c2, c3, …).

For example, the reference point 10 may include at least one of a mark nail, a buried pin, a mark pole, a mark pin, a manhole, and a mark nail exposed to the ground while being linked to an underground facility, and such a reference point 10 The role indicated by the reference point 10 may be indicated by a first code c1, for example, a first code c1 indicating a water supply, a sewer, a sewage pipe, a street light line, an underground transmission line or a communication line, etc. It can be marked with an intaglio or weak-engraving on a metal sign nail.

In addition, a second code (c2) indicating an area in which the reference point 10 is installed may be displayed at the reference point 10. For example, a second code c2 indicating an area name, an address name, or a road address name may be displayed. It can be marked with an intaglio or embossing on a sign nail made of metal.

In addition, a third code (c3) indicating the pipe diameter or pipe direction of the pipe connected to the reference point 10 may be displayed on the reference point 10. For example, a third code (c3) indicating a pipe diameter value or an arrow indicating a pipe direction may be displayed. 3 The code (c3) may be marked in an intaglio or embossed mark on a metal mark.

At this time, the terminal 100 interlocks with a deep-learning neural network installed in the terminal 100, the content providing server 200 and/or an external server in order to perform image deep learning. I can. Here, the deep learning neural network is at least one deep learning of a convolutional neural network (CNN), regions with CNN features (R-CNN), Fast R-CNN, Faster R-CNN, and Mask R-CNN. It may include a neural network.

Next, the terminal 100 may detect an ellipse from the detected reference point 10 object. (S205) In detail, the reference point 10 is a circular sign on the road such as a sign nail, and may be recognized as an elliptical object in the photographed image by the photographing direction.

Accordingly, the terminal 100 may first perform an ellipse detection algorithm using "Projective Invariant Pruning" in order to more clearly recognize the detected reference point 10.

In detail, the circular reference point 10 is often recognized as an ellipse depending on the direction in which the image is taken. However, when the reference point 10 is recognized as an ellipse, it may be difficult to accurately detect a code assigned to the reference point 10. Therefore, in the embodiment of the present invention, the code of the reference point 10 is accurately detected by applying an ellipse detection algorithm using "Projective Invariant Pruning", and the overall shape is covered even if the edge of the ellipse of the reference point 10 is partially covered. Can be detected.

7 is an example of a reference point 10 according to an embodiment of the present invention, and is a diagram for explaining a method of automatically extracting an effective area from the reference point 10.

In more detail, referring to FIG. 7, the terminal 100 may first detect an ellipse from the reference point 10.

In this case, the terminal 100 may perform a processing algorithm for removing a false positive among the detected ellipses. Here, a false positive is defined as an error that judges that there is something that does not exist by selecting the wrong ellipse.

That is, the circular reference point 10 has a product with a border. In this case, the terminal 100 recognizes arcs of different areas in the ellipse detection and removes the detection of unwanted ellipses. The processing algorithm for can be applied.

8 is an exemplary diagram for selecting an effective elliptical region through an Intersection Of Union (IOU) according to an embodiment of the present invention.

Referring to FIG. 8, the ellipse recognition rate score is proportional to the overlapping arc, and it is necessary to remove the ellipse overlapping the ellipse with a high score. The method of automatically selecting the ellipse area was applied. For reference, an ellipse with a score of Intersection Of Union (IOU) greater than a predetermined setting value may be selected, or an ellipse may be selected through relative comparison.

That is, the terminal 100 recognizes at least one or more elliptical regions in automatically extracting the region in which the code is displayed from the reference point 10 of the captured image, and an effective ellipse through the Intersection Of Union (IOU). Area can be automatically selected.

9 is a view for explaining a process of performing a homography conversion in order to convert an elliptical reference point 10 into a circular reference point 10 according to an embodiment of the present invention.

In addition, referring to FIG. 9, the terminal 100 automatically selects an effective ellipse region based on at least one ellipse region through Intersection Of Union (IOU), and then homography to remove geometric distortion of the ellipse. ) By performing the transformation, it is possible to obtain a circle-shaped reference point 10 object and homography transformation information. (S207)

That is, the terminal 100 may transform the elliptical reference point 10 image into a circular reference point 10 image by performing a homography transformation on the reference point 10, and through this, the reference point 10 ) By acquiring an image in the form of viewing the object vertically from the top, the code and/or the size of the corresponding reference point 10 can be more accurately recognized.

In this case, the terminal 100 may obtain homography conversion information through a homography conversion process.

Here, the homography conversion information is information on the change elements generated in the object of the reference point 10 in the process of converting the reference point 10 from an elliptical to a circle shape (e.g., inclination change information of a circle, size change information, It may mean a change information of a horizontal diameter or information about a change of a vertical diameter). This homography conversion information may be used as base data for obtaining an initial camera posture of the terminal 100 later.

In addition, the terminal 100 may recognize a code assigned to the reference point 10 by deep learning the object of the reference point 10 having a circular shape. (S209)

In detail, the terminal 100 performs image deep learning (Identification) based on the garden-shaped reference point 10 to specify the reference point 10 and the underground facility matching the reference point 10 Can be detected.

In this case, in an embodiment, the terminal 100 may perform image deep learning for code detection through a convolution neural network (CNN), and the reference point 10 through a convolution neural network. The code assigned to the reference point 10 may be detected by extracting the feature map and deriving the code assigned to the reference point 10 based on the extracted feature map.

At this time, the terminal 100 may detect at least one or more of a plurality of codes through image deep learning for the reference point 10 object, and later match the reference point 10 based on the detected at least one or more codes. 3D spatial model information can be accurately acquired.

Subsequently, the terminal 100 that recognizes the code based on the circle-shaped reference point 10 object may acquire 3D spatial model information matching the recognized code. (S211)

In detail, the terminal 100 may search the database 240 to obtain 3D spatial model information matching the recognized code. Here, in this embodiment, it has been described that the terminal 100 obtains 3D spatial model information by searching the database 240, but it is also possible to obtain 3D spatial model information matching the code recognized in the memory 160. Etc. Various embodiments are possible. However, it will be the most preferable embodiment to obtain 3D spatial model information from the database 240, which is more suitable for managing a vast amount of data.

That is, the terminal 100 acquires 3D spatial model information matching the recognized code, and the 3D coordinates for the reference point 10 (for example, coordinates indicating the absolute position of the reference point 10), the reference point It is possible to obtain information on underground facilities matching the scale of (10) and the reference point (10).

In this case, the size of the reference point 10 may be information further including the number of pixels indicated by the reference point 10.

For example, referring to FIG. 10, the terminal may recognize a code assigned to a corresponding sign by deep learning a garden-shaped sign nail object, and the database 240 based on the identification information for the recognized code By searching, information on a 3D spatial model matching the corresponding cover nail may be obtained.

Next, the terminal 100 may set 3D spatial information based on the obtained homography transformation information and 3D spatial model information. (S213)

Here, the 3D spatial information is information that provides position and orientation related information based on the viewpoint of the terminal looking at the 3D space, and is matched to the initial camera posture, 3D spatial coordinate system, initial camera position, and 3D spatial coordinate system. It may include underground facility information and/or reference point scale information.

First, the terminal 100 may acquire an initial camera posture based on the obtained homography transformation information and the 3D coordinates of the reference point 10.

Here, the initial camera posture may be information indicating the posture (ie, azimuth) of the camera 150 when the camera 150 of the terminal 100 starts photographing a 3D space including the reference point 10 (initial). have.

That is, the initial camera posture is information obtained by obtaining coordinate values for Pitch, Yaw, and Roll coordinates, which are direction tracking coordinate systems based on the 3D coordinates of the reference point 10 in the 3D space. It may be orientation information that determines in which direction the camera 150 of the terminal 100 is facing.

Here, among the direction tracking coordinate systems in the 3D space, the pitch coordinate system can detect the x-axis rotation of the terminal 100, that is, the vertical rotation, and the yaw coordinate system detects the y-axis rotation of the terminal 100, that is, the left and right rotation. The roll coordinate system may detect the z-axis rotation of the terminal 100, that is, the forward and backward rotation.

Returning again, in detail, the terminal 100, in the process of converting the reference point 10 from an elliptical to a circular shape, the change factor information generated in the reference point 10 object (e.g., the inclination change information of the circle, the horizontal diameter) The initial posture by the camera 150 of the terminal 100 based on the homography transformation information, which means change information, vertical diameter change information, etc.), and the three-dimensional coordinates of the reference point 10 included in the three-dimensional spatial model information (Azimuth) can be estimated, and through this, an initial camera posture can be obtained.

In addition, the terminal 100 may set a 3D spatial coordinate system capable of tracking the position of the terminal in 3D space based on the 3D coordinates of the reference point 10 of the acquired 3D spatial model information.

Here, the three-dimensional space coordinate system may be generated based on the reference point 10 by matching one for one three-dimensional space, and the three-dimensional coordinates of the reference point 10 implemented by the x-axis, y-axis and z-axis It may be generated as an extension line of or may be generated by designating at a specific location in a 3D space based on the 3D coordinates of the reference point 10 according to a predetermined method.

In addition, the terminal 100 may obtain information on underground facilities matching the 3D spatial coordinate system based on the set 3D spatial coordinate system and information on the underground facilities matching the reference point 10 of the 3D spatial model information.

In detail, the terminal 100 searches the memory 160 and/or the database 240 based on the information on the underground facilities matching the reference point 10 and the 3D spatial coordinate system, and information on the underground facilities matching the 3D spatial coordinate system. Can be obtained.

In addition, the terminal 100 may acquire information on underground facilities matching the 3D spatial coordinate system and use it to match a virtual object representing the underground facilities in the 3D space in the future.

In addition, the terminal 100 may obtain reference point scale information based on the size of the reference point 10 of the 3D spatial model information acquired based on the reference point 10 and pixel information displaying the reference point 10 object. I can.

Here, the reference point scale information is displayed on the captured image based on a preset size of the reference point 10 (eg, a diameter or radius of the circular reference point 10) and a pixel for displaying the size of the reference point 10 It may be information for determining the scale of the reference point 10 to be used.

That is, the terminal 100 may acquire the reference point scale information as described above and perform a process of adjusting the scale of the virtual object according to the reference point 10 scale in the future.

In addition, the terminal 100 may set an initial camera position based on the acquired 3D spatial coordinate system and reference point scale information.

In detail, the terminal 100 may first obtain distance information between the reference point 10 and the terminal according to the size of the reference point 10 based on the reference point scale information.

Here, the reference point 10-terminal distance information may be information obtained by calculating a physical distance between the reference point 10 photographed through the camera 150 of the terminal 100 and the corresponding terminal 100.

In detail, the terminal 100 acquires information on the scale of the reference point 10, that is, the size (eg, the diameter or radius of the circular reference point 10) in the image captured by the camera 150 based on the reference point scale information. The distance between the terminal 100 that photographs the reference point 10 and the reference point 10 may be calculated according to the obtained scale of the reference point 10.

At this time, the distance between the reference point 10 and the terminal 100 according to the scale of the reference point 10 in the captured image is a predetermined reference (e.g., a distance from the preset terminal 100 according to the diameter of the reference point 10, etc.) It may be preset according to, and may be pre-stored in the memory 160 of the terminal 100 and/or the database 240 of the content providing server 200.

For example, when the diameter of the reference point 10 of the captured image on the memory 160 is 1 cm or more and less than 3 cm, the reference point 10-terminal 100 is 50 cm, 3 cm or more and 5 cm or less. If the distance between the reference point 10 and the terminal 100 is 30 cm, and less than 5 cm and 7 cm, the distance between the reference point 10 and the terminal 100 may be 10 cm. In addition, when it is determined that the diameter of the reference point 10 of the captured image is 5 cm through the reference point scale information, the terminal 100 may determine the distance between the reference point 10 and the terminal 100 as 10 cm, and the reference point ( 10)- The terminal distance information can be determined as 10cm.

In addition, the terminal 100 may determine the location of the terminal 100 based on the 3D spatial coordinate system based on the acquired reference point 10-terminal distance information, and the determined location of the terminal 100 Based on the initial camera position can be set.

As described above, in the embodiment of the present invention, the terminal 100 may convert 3D spatial information set based on the reference point 10 into 3D spatial information based on the terminal 100 as described above.

For example, the terminal 100 may obtain a 3D spatial coordinate system set based on the location of the terminal 100 and/or information on underground facilities matching the 3D spatial coordinate system and use it as 3D spatial information. have.

That is, the terminal 100 may acquire 3D spatial information in various ways to implement a mixed reality providing service related to underground facilities according to an embodiment of the present invention.

To this end, the terminal 100 having set 3D spatial information as described above may acquire scene specification information based on the set 3D spatial information. (S215)

Here, the scene specification information is information that detects a virtual object matching a 3D space photographed by the camera 150 of the terminal 100, and assists in matching the detected virtual object with the corresponding 3D space. It may include at least one or more of object position (coordinate) setting information, virtual object orientation setting information, and virtual object scale setting information.

In detail, the terminal 100 is based on the 3D spatial coordinate system of the acquired 3D spatial information and the underground facility information matching the 3D spatial coordinate system, the location of the virtual object acquired in the 3D space of the captured image ( Coordinates) can be set, and virtual object location (coordinates) setting information can be created.

For example, the terminal 100 may acquire the location of the virtual object in the 3D space of the captured image based on the 3D spatial coordinate system and the absolute position information of the underground facility information matching the 3D spatial coordinate system. And, through this, it is possible to generate the virtual object location (coordinate) setting information.

In addition, the terminal 100 may set the orientation of the acquired virtual object for the 3D space of the captured image based on the initial camera posture of the acquired 3D spatial information, and may generate the virtual object orientation setting information. .

That is, the terminal 100 can determine the direction (orientation) of the viewpoint that the terminal 100 has in the 3D space, based on the orientation of the terminal 100 obtained through the initial camera posture, Virtual object orientation setting information may be generated so that the viewpoint direction of the terminal 100 and the orientation of the virtual object match.

In addition, the terminal 100 may set the scale of the acquired virtual object for the 3D space of the captured image based on the reference point scale information of the acquired 3D spatial information, and generate the virtual object scale setting information. can do.

Here, the virtual object scale setting information may be information set so that the size of the virtual object is displayed in a 3D space of the captured image in proportion to the size of the reference point 10.

In detail, the terminal 100, first, a pixel for displaying the size of the reference point 10 preset through reference point scale information (eg, the diameter or radius of the circular reference point 10) and the size of the reference point 10 It is possible to obtain the number of information.

In addition, the terminal 100 may detect a pixel on the same line as the location of the reference point 10 in the 3D space, that is, a plurality of pixels representing the same distance as the reference point 10.

In this case, when displaying a virtual object based on a plurality of pixels that display the same distance as the detected reference point 10, the terminal 100 has a size proportional to the size of the reference point 10 and the number of pixels and the number of pixels. Virtual object scale information can be generated so that the corresponding virtual object is displayed as a number.

For example, the terminal 100 may estimate the number of pixels according to the size of the reference point 10 in meters (m), and based on this, may determine the number of pixels according to the size of the virtual object.

In detail, for example, the terminal 100 may be preset to estimate 0.01 meters as 1 pixel, and it may be confirmed that the size of the reference point 10 is 0.1 meters based on the size information of the preset reference point 10. have. In addition, the terminal 100 may determine that the reference point 10 is output based on 10 pixels on the 3D space according to a preset metric estimation principle.

That is, when the size of the reference point 10 is 0.1 meters, the terminal 100 may obtain reference point scale information indicating that the image is displayed based on 10 pixels in an image outputting a 3D space.

At this time, when the terminal 100 displays a virtual object based on a plurality of pixels representing the same distance as the reference point 10, the predetermined size of the virtual object is based on the underground facility information for the virtual object. It can be seen that is, for example, 1 meter.

And the terminal 100, in proportion to the number of pixels according to the size of the acquired reference point 10, that is, 10 pixels at 0.1 meter, the number of pixels according to the size of the corresponding virtual object 1 meter to 100 pixels. It can be determined, and based on this, virtual object scale information can be generated.

That is, the terminal 100, the virtual object scale information proportional to the reference point scale information (the size of the reference point 10: 0.1 meters, the number of display pixels: 10 pixels) (virtual object size: 1 meter, the number of display pixels: 100 pixels) ) Can be set so that the virtual object is displayed in a scale corresponding to the background image in the 3D space.

In addition, after setting the virtual object scale for a plurality of pixels representing the same distance as the reference point 10 based on the reference point scale information, the terminal 100 , The number of pixels according to the size of the virtual object may be determined according to a predetermined degree of perspective based on the virtual object scale set for pixels on the same distance line as the reference point 10.

For example, the terminal 100, when the virtual object scale information for a plurality of pixels displaying the same distance as the reference point 10 is set to'virtual object size: 1 meter, number of display pixels: 100 pixels', the corresponding For a plurality of pixels that display a virtual object for a far distance than a distance, the number of pixels displayed can be set to gradually decrease to less than 100 pixels as the distance increases. As for the distance, the number of display pixels may be set to increase gradually so that the number of display pixels exceeds 100 pixels.

That is, the terminal 100 determines the scale of the virtual object based on the scale information of the reference point 10, and corresponds to the real objects (in the embodiment, the reference point 10) that exist in the three-dimensional space of the captured image. The virtual object can be augmented and displayed in a size that is set.

Next, the terminal 100 that has obtained the scene specification information may render a virtual object based on the obtained scene specification information and display it as a mixed reality (MR) image. (S217)

11 is an example of displaying a rendered virtual object on a captured image according to an embodiment of the present invention.

In detail, referring to FIG. 11, the terminal 100 matches virtual objects matched on the 3D space of the captured image based on the acquired scene specification information, the initial camera posture and the initial camera position of the 3D spatial information. You can render.

In more detail, the terminal 100 may determine the viewpoint of the camera 150 of the terminal 100, that is, the attitude (orientation) and position of the terminal, based on the acquired initial camera posture and the initial camera position, and scene specification information Rendering may be performed by matching virtual objects matching the 3D space of the captured image according to the viewpoint of the camera 150 determined based on the determination.

In this case, the terminal 100 may superimpose and display a virtual object matched and rendered in a 3D space on an image captured by the camera 150 through the display unit 120.

In detail, in order to superimpose the virtual object on the three-dimensional space of the captured image with an accurate position, the terminal 100 first selects coordinates previously allocated to each real object existing in the three-dimensional space based on the three-dimensional space coordinate system. It can be derived from the memory 160 and/or the database 240.

In addition, the terminal 100 may match and display a real object located in a 3D space and each virtual object to be matched to the corresponding 3D space based on the derived coordinates for each real object.

In addition, the terminal 100 may impart a predetermined transparency to a virtual object that is superimposed and displayed in a 3D space so that the virtual object can be naturally superimposed and displayed on the captured image.

In addition, the terminal 100 may match and store the positional relationship between the virtual object rendered as above and the real object of the captured image three-dimensionally, and through this, the shooting time point by the camera 150 of the terminal 100 Even if it is changed, the virtual object can be accurately matched and displayed on the real object by moving the virtual object according to the position of the real object according to the change in the shooting time.

In detail, the terminal 100 may update the scene specification information by tracking changes in the position and orientation (position) of the camera 150 of the terminal 100, and a photographed image based on the updated scene specification information. It is possible to perform augmented rendering to match virtual objects matched to the 3D space of, and display the rendered virtual objects by superimposing them on the captured image. (S219)

In more detail, the terminal 100 may track changes in the position and orientation of the camera 150 of the terminal 100 based on the inertial sensor (IMU, 140), and the camera of the terminal 100 obtained through tracking ( 150) An update may be performed with scene specification information for a 3D space matched based on a change in position and orientation.

In addition, the terminal 100 may render a virtual object matching the three-dimensional space of the captured image based on the updated scene specification information, and display the superimposed display on the captured image by the camera 150 through the display 120. have.

At this time, the terminal 100 3D matches the positional relationship between the augmented-rendered virtual object and the real object of the captured image based on the initial camera posture and the initial camera position, and based on the previously stored information, the terminal 100 ), the virtual object can be moved according to the position of the real object according to the change in the shooting time when the shooting time is changed by the camera 150 of the terminal 100, and through this, even when the position and orientation of the camera 150 of the terminal 100 is changed, the virtual object Objects can be accurately and conveniently aligned and displayed in a three-dimensional space.

That is, the terminal 100 may match the position of a real object and a virtual object in a three-dimensional space to display a virtual object superimposed on the real object. In this case, the user can apply a predetermined transparency to the virtual object. The real object behind the virtual object can be recognized together, and even if the shooting time point by the camera 150 of the terminal 100 is changed, the virtual object can be moved to fit the position of the changed real object and displayed accurately superimposed. .

Meanwhile, FIG. 12 is a flowchart illustrating a method of providing mixed reality related to underground facilities based on GPS linkage according to another embodiment of the present invention.

Referring to FIG. 12, in another embodiment of the present invention, the terminal 100 may provide a mixed reality related to an interactive underground facility by tracking changes in the location and posture of the terminal 100. Hereinafter, for effective description, descriptions overlapping with those described above may be summarized or omitted.

In detail, first, the terminal 100 may acquire a photographed image and 3D spatial information by controlling the camera 150. (S301)

Here, the terminal 100 may acquire 3D spatial information based on the reference point 10 according to the above-described embodiment.

For example, referring to FIG. 13, the terminal 100 uses a mark nail installed in the field as a reference point 10, and a mark nail object from a photographed image acquired through the camera 150 according to the above-described embodiment. May be detected, and 3D spatial information may be obtained based on the detected target nail object.

In another embodiment, the terminal 100 may acquire a three-dimensional spatial coordinate system of three-dimensional spatial information using DGPS (high-precision GPS). That is, the terminal 100 may acquire a 3D spatial coordinate system capable of tracking the location of the terminal in the 3D space of the captured image based on high precision GPS (DGPS).

Next, the terminal 100 may display a mixed reality (MR) image according to the acquired 3D spatial information. (S303)

In detail, the terminal 100 may acquire scene specification information for a corresponding 3D space based on the acquired 3D spatial information, and based on the acquired scene specification information and 3D spatial information, A virtual object matching to can be rendered. In addition, the terminal 100 may display the rendered virtual object as a mixed reality (MR) image by superimposing the rendered virtual object on a captured image representing a corresponding three-dimensional space.

In addition, the terminal 100 may track changes in the position and orientation of the camera 150 according to the movement and posture of the terminal 100. (S305)

In detail, the terminal 100 first updates the 3D spatial information by tracking changes in the position and orientation of the camera 150 of the terminal 100 based on the acceleration sensor and/or the gyro sensor of the inertial sensor (IMU, 140). can do.

Here, the sensitivity to the change in the position and orientation of the inertial sensor (IMU, 140) is very high, and it is possible to immediately sense the position and orientation change information of the terminal 100 that changes minutely in real time, and the processor Based on real-time sensing data of the IMU 140, a change in the location and orientation of the terminal 100 may be tracked to update and display a virtual object.

However, as the tracking information of the inertial sensor (IMU, 140) accumulates, there is a possibility that the accuracy of the spatial tracking accumulative error occurs and thus the accuracy decreases, and this may make it difficult for the virtual object to be displayed in an appropriate position.

In order to prevent this, the terminal 100 can track the position change of the camera 150 based on DGPS (high-precision GPS), and the three-dimensional spatial coordinate system tracked by the inertial sensor (IMU, 140) in real time. By updating, the accumulated error can be corrected.

In addition, in the embodiment, the terminal 100 is based on the real object acquired through the camera 150, that is, the reference point 10, based on the information recognized by the visual inertial odometer (VIO, Visual Inertial Odometry) technology ( 150) posture and position change may be tracked, and 3D spatial information may be updated based on this, thereby correcting a cumulative error of 3D spatial information tracked by the inertial sensor (IMU, 140).

In detail, in the present embodiment, the terminal 100 uses a visual inertial odometer (VIO) technology as shown in FIG. 14 in order to more accurately track changes in the attitude and position of the camera 150 to update 3D spatial information. Based on the movement of the terminal 100, a change in the position and orientation of the camera 150 may be tracked.

Here, the visual inertial rangefinder technology refers to a technology for determining a location and a direction based on a terrain feature by analyzing visual information acquired through the camera 150.

That is, the terminal 100 uses the recognition information on the reference point 10 (eg, a cover nail and/or a manhole, etc.), which is a real object acquired from the camera 150, the position and orientation of the camera 150. The change may be determined, and the 3D spatial information may be updated by correcting the tracking information acquired through the inertial sensor (IMU, 140) and the DGPS (high-precision GPS) based on this.

In more detail, first, the terminal 100 may detect an object of the reference point 10 in a 3D space acquired through an image captured by the camera 150.

That is, the terminal 100 may perform image deep learning based on the captured image to detect an object of the reference point 10 that is matched with each underground facility and installed on the site, among real objects identified from the captured image.

In addition, the terminal 100 may update 3D spatial information based on at least one or more reference points 10 detected in the captured image.

In detail, the terminal 100 may perform location and orientation change tracking of the camera 150 based on the visual inertial rangefinder technology based on the detected at least one reference point 10.

In this embodiment, the method of tracking the position and orientation change of the camera 150 based on the visual inertial rangefinder technology may apply a conventionally known mathematical algorithm, and in the present invention, the camera based on the visual inertial rangefinder technology ( 150) does not limit or limit the algorithm itself that performs location and orientation change tracking.

That is, the terminal 100 may perform location and orientation change tracking of the camera 150 based on the VIO technology based on the sensed at least one reference point 10, and through this, the space-camera 150 It is possible to generate tracking correction information based on location and orientation tracking.

Here, the tracking correction information may be information for correcting tracking information obtained by the inertial sensor (IMU, 140) and/or a high-precision GPS (DGPS) based on the tracking information obtained based on VIO technology.

For example, the tracking correction information is the reference point 10 of the captured image according to the position and orientation of the camera 150 obtained by the inertial sensor (IMU, 140) and/or DGPS (high-precision GPS) (e.g. ) It may be information including error information between the position and the position of the reference point 10 (eg, a mark nail) of the captured image according to the position and orientation of the camera 150 obtained by the visual inertial rangefinder technology.

In detail, for example, the terminal 100 includes a first reference point in the captured image of the camera 150 according to the position and orientation of the camera 150 acquired by the inertial sensor (IMU, 140) and/or DGPS (high-precision GPS). The position of the second reference point may be obtained based on a 3D matched coordinate system matched to a corresponding 3D space. In addition, the terminal 100 matches the positions of the first reference point and the second reference point in the image captured by the camera 150 according to the position and orientation of the camera 150 acquired by the visual inertial rangefinder technology to a corresponding 3D space. It can be obtained on the basis of the three-dimensional matched coordinate system.

And the terminal 100, the position of the first and second reference points 10 by the inertial sensor (IMU, 140) and / or DGPS (high-precision GPS) and the first and second reference points 10 by the visual inertial rangefinder technology. By comparing the locations, it is possible to determine whether they match exactly.

At this time, the terminal 100, the position of the first and second reference points 10 by the inertial sensor (IMU, 140) and / or DGPS (high-precision GPS) and the first and second reference points 10 by the visual inertial rangefinder technology By grasping the degree of matching between the positions of, error information indicating the degree of the error between the positions may be obtained.

Subsequently, the terminal 100 that has obtained the error information may generate tracking correction information based on the obtained error information, and based on the generated tracking correction information, an inertial sensor (IMU, 140) and/or DGPS ( It is possible to correct tracking information for a change in the position and orientation of the camera 150, which has been generated by high-precision GPS).

In addition, the terminal 100 correcting the tracking information based on the visual inertial rangefinder technology may update the 3D spatial information based on the corrected tracking information.

In the above, the position of the reference point 10 of the photographed image according to the position and orientation of the camera 150 obtained by the inertial sensor (IMU, 140) and/or DGPS (high-precision GPS), and the position of the reference point 10 obtained by the visual inertial rangefinder technology. Limited to the example of generating tracking correction information based on the error between the position of the reference point 10 of the captured image according to the position and orientation of the camera 150, and correcting the existing tracking information based on this, and updating the 3D spatial information Although described above, this is only an example of the present invention and is not limited thereto, and tracking the position and orientation of the camera 150 based on the inertial sensor (IMU, 140) and/or DGPS (high-precision GPS) and VIO technology, and 3 Various embodiments of updating dimensional spatial information may be possible.

Next, the terminal 100, which tracks the change in the position and orientation of the camera 150 according to the movement of the terminal, and updates the 3D spatial information based on this, is based on the 3D spatial information updated according to the tracking. The scene specification information can be updated. (S307)

In detail, the terminal 100 may perform an update with scene specification information for a 3D space matched based on the updated 3D spatial information.

That is, the terminal 100 may update the scene specification information generated based on the initial 3D spatial information based on the updated 3D spatial information.

In detail, the terminal 100 updates at least one of the virtual object position (coordinate) setting information, the virtual object orientation setting information, and the virtual object scale setting information of the scene specification information based on the updated 3D spatial information. can do.

Next, the terminal 100 having updated the scene specification information may display the rendered mixed reality (MR) image according to the updated scene specification information. (S309)

In detail, the terminal 100 matches the virtual objects matched on the 3D space of the captured image based on the updated scene specification information and the initial camera posture and the initial camera position of the updated 3D spatial information and rendering. )can do.

In more detail, the terminal 100 may determine the viewpoint of the camera 150 of the terminal 100, that is, the attitude (orientation) and position of the terminal based on the initial camera posture and the initial camera position of the updated 3D spatial information. In addition, the virtual object matching the 3D space of the captured image according to the viewpoint of the camera 150 determined based on the updated scene specification information may be matched to perform rendering.

In addition, the terminal 100 may output the rendered virtual object by superimposing it on the captured image through the display unit 120.

As described above, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention integrates data related to underground facilities and mixed reality (MR) technology in connection with the user's terminal to facilitate management and management of underground facilities. By supervising, based on mixed reality technology, real-time information on underground facilities such as water and sewage, electricity, communication, gas pipes, building electrical wiring, and heating and cooling pipes can be visualized and provided to users in 3D. There is an effect of building a mixed reality service infrastructure that provides empirical contents specialized for landfill inspection and maintenance.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention, by interacting with the user's terminal in real time, are updated in real time according to the change in the location and/or orientation (position) of the user terminal. By providing the information using mixed reality, there is an effect that the user can accurately and conveniently provide information on the underground facilities matching the space in which the user wants to check the information on the underground facilities.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention have improved accuracy by matching and matching 3D space and underground facilities using DGPS (Differential Global Positioning System, high-precision GPS). It has the effect of providing information on underground facilities.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention implements image recognition/tracking/learning technology and 3D space recognition/camera tracking technology for providing information on underground facilities, thereby providing high-quality underground facilities. Facility information can be provided, and through this, there is an effect of maximizing the efficiency of underground facility investigation, analysis, design, education prevention, and supervision work.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention can use a camera and/or an inertial sensor (IMU) of the terminal to determine the location and/or orientation (posture) of the user terminal. By providing information on underground facilities according to changes, it is possible to provide information on underground facilities according to changes in the state of the user terminal through its own algorithm and process even without a separate location tracking device.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention provides an underground facility buried underground based on 3D spatial information provided by the national spatial information platform (V-WORLD) on an electronic map drawing. By mapping, it is possible to provide a more realistic and accurate shape of underground facilities including digital terrain, and there is an effect of ensuring compatibility and performance of existing infrastructure and new software (S/W).

In addition, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded in the computer-readable recording medium may be specially designed and constructed for the present invention or may be known and usable to those skilled in the computer software field. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magnetic-optical media such as floptical disks. medium), and a hardware device specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device can be changed to one or more software modules to perform the processing according to the present invention, and vice versa.

Specific implementations described in the present invention are examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings exemplarily represent functional connections and/or physical or circuit connections, and in an actual device, various functional connections that can be replaced or additionally It may be referred to as a connection, or circuit connections. In addition, if there is no specific mention such as “essential” or “importantly”, it may not be an essential component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to a preferred embodiment of the present invention, the spirit of the present invention described in the claims to be described later if one of ordinary skill in the relevant technical field or those of ordinary skill in the relevant technical field And it will be understood that various modifications and changes can be made to the present invention within a range not departing from the technical field. Therefore, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be determined by the claims.

Claims (7)

As a method of providing mixed reality related to underground facilities that manages and supervises facilities buried in the basement based on data related to underground facilities and mixed reality technology in the processor of the terminal,
Controlling a camera to obtain a photographed image;
Detecting a reference point object by image processing the captured image;
Recognizing a code displayed at a reference point in the reference point object;
Obtaining 3D spatial model information matching the recognized code;
Setting 3D spatial information for the captured image based on the acquired 3D spatial model information; And
And displaying a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the 3D spatial information.
A method of providing mixed reality related to underground facilities. The method of claim 1,
The acquiring 3D spatial model information matching the recognized code includes acquiring 3D coordinates of the reference point object specifying a 3D space of the captured image.
A method of providing mixed reality related to underground facilities. The method of claim 2,
The step of setting 3D spatial information for the captured image includes setting a 3D spatial coordinate system for the captured image based on the 3D coordinates of the acquired reference point object.
A method of providing mixed reality related to underground facilities. The method of claim 2,
The step of obtaining 3D spatial model information matching the recognized code further comprises obtaining a scale of the reference point object,
The step of setting 3D spatial information for the captured image includes obtaining the reference point scale information based on a size of the reference point object and a pixel displaying the reference point object.
A method of providing mixed reality related to underground facilities. The method of claim 4,
In the step of displaying a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the 3D spatial information, the size of the virtual object matching the 3D space of the captured image is scaled by the reference point. Comprising the step of augmenting the captured image according to the information
A method of providing mixed reality related to underground facilities. The method of claim 1,
The step of updating the 3D spatial information by tracking changes in the position and orientation of the camera, and rendering and displaying a virtual object updated according to the updated 3D spatial information on the captured image. doing
A method of providing mixed reality related to underground facilities. The method of claim 1,
The updating of the 3D spatial information by tracking changes in the position and orientation of the camera includes: updating the 3D spatial information based on real-time sensing data of an inertial sensor (IMU), and the updated Comprising the step of correcting the updated 3D spatial information based on the position and direction of a reference point object recognized based on a visual inertial odometry (VIO) technology for 3D spatial information.
A method of providing mixed reality related to underground facilities.

Images