KR102264219B1 - Method and system for providing mixed reality contents related to underground facilities - Google Patents

Abstract

The method for providing mixed reality related to underground facilities according to the embodiment is a method of providing mixed reality related to underground facilities in which a processor of a terminal manages and supervises facilities buried underground based on data related to underground facilities and mixed reality technology, and a camera controlling to obtain a captured image; detecting a reference point object by image processing the captured image; recognizing a code displayed on a reference point in the reference point object; obtaining 3D spatial model information matching the recognized code; setting 3D spatial information for the captured image based on the obtained 3D spatial model information; and displaying a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the 3D spatial information.

Description

METHOD AND SYSTEM FOR PROVIDING MIXED REALITY CONTENTS RELATED TO UNDERGROUND FACILITIES

The present invention relates to a method and system for providing mixed reality (MR) related to underground facilities. More specifically, it relates to a method and system for providing a mixed reality related to an underground facility for managing and supervising facilities buried underground by fusion of data related to underground facilities and mixed reality technology in conjunction with a terminal.

In general, underground facilities, which are defined as facilities buried underground, such as water supply facilities, sewage facilities, gas facilities, communication facilities, and electric facilities, are essential infrastructure for operating a city. It is not only difficult to manage due to the variety of accidents, but also has the characteristic that it is difficult to derive countermeasures in case of an accident.

In particular, in the case of an unexpected accident with an underground facility, it can be said that systematic management is very important as it causes loss of life or property.

Accordingly, it provides an information recognition means that can easily detect the movement of underground facilities even when natural disasters such as earthquakes, landslides and floods occur by installing a ground level surface above ground or installing an underground level surface when underground facilities are buried. Various management systems have been proposed.

In the conventional management system, a location corresponding to an underground facility is recognized through a global positioning system (GPS), or an information recognition means such as a barcode or a QR code is installed on the ground and an underground facility matched with the information recognition means By building a database with related data, the location of underground facilities is recognized.

Thereafter, the field worker or manager acquires location information and attribute information through tag recognition or wireless communication method tag recognition for the above-described GPS location information and information recognition means using the terminal, and searches the database using this. By doing so, it is possible to receive the stored underground facility information and perform management work on the underground facility.

However, according to the currently provided information on underground facilities, it remains at the level of displaying only the location and approximate shape of the underground facility by simply mapping the corresponding underground facility on the map, so it is possible to determine the three-dimensional shape of the facilities in the underground space. There is a limitation in that it is difficult to identify easily.

In addition, if an underground facility is mapped based on the location recognized through the satellite navigation system, which inevitably causes an error, the location of the underground facility cannot be accurately displayed due to the error, making it difficult to perform management work on the underground facility. There is a problem, and there is a problem that it is difficult to use in an urban area where the satellite navigation system does not work well.

In addition, there is a problem in that it is difficult to separately install information recognition means such as barcodes or QR codes on the ground one by one, and if the information recognition means exposed on the ground is damaged, there is a problem of becoming useless.

In addition, the conventional management system maps and provides pre-stored underground facility information on a map, but interacts with information related to underground facilities in consideration of real-time changes in the location and/or orientation (attitude) of the user terminal using the underground facility information. There is insufficient to provide (interaction).

In particular, in the absence of GPS, the conventional management system has difficulty in accurately providing information about underground facilities interacted with according to a change in the user's location and/or the camera orientation (posture) of the terminal, so technical introduction is required to solve this problem. .

In addition, in the electronic map provided by private companies, the locations of objects such as terrain and buildings appearing on the map are set in relative coordinates, so the distance between underground facilities and each other's azimuth angles do not match in the coordinates. case often occurs.

In addition, the underground facility information stored in the database of the conventional underground facility management system is requested by a private company, or the location information, topographic information, and attribute information of the underground facilities are mapped on the 2D or 3D electronic map built by the company itself. There is a limit in that the accuracy is significantly low as it is manufactured in this way.

On the other hand, recently, when shooting using a camera module, the virtual world is grafted onto the real world to provide an environment in which real physical and virtual objects can interact, providing content using the Mixed Reality (MR) technique. is being studied.

In detail, mixed reality (MR) includes the meaning of augmented reality (AR: Augmented 　 Reality) that adds virtual information based on reality and augmented virtual (AV: 　 Augmented 　 Virtuality) that adds real information to a virtual environment. By providing a smart environment in which virtual and virtual are naturally connected, users can have a rich experience.

10-2018-0132183 A

The present invention has been devised to solve the above problems, and it is related to underground facilities that easily manage and supervise underground facilities by fusion of data related to underground facilities and mixed reality (MR, Mixed Reality) technology in conjunction with the user's terminal. An object of the present invention is to provide a method and system for providing mixed reality.

In detail, an embodiment of the present invention has an object to provide information about underground facilities according to a change in the location and/or orientation (attitude) of the user terminal by using a mixed reality by interacting with the user's terminal in real time.

In addition, an embodiment of the present invention aims to provide a method and system for providing an underground facility-related mixed reality that implements image recognition/tracking/learning technology and 3D spatial recognition/camera tracking technology for providing information on underground facilities. have.

In addition, an embodiment of the present invention provides an underground facility-related mixed reality that easily provides information about underground facilities according to a change in the location and/or orientation (attitude) of a user terminal in a mixed reality even in the absence of a global positioning system (GPS). An object of the present invention is to provide a providing method and system.

However, the technical problems to be achieved by the present invention and embodiments of the present invention are not limited to the technical problems as described above, and other technical problems may exist.

The method for providing mixed reality related to underground facilities according to the embodiment is a method of providing mixed reality related to underground facilities in which a processor of a terminal manages and supervises facilities buried underground based on data related to underground facilities and mixed reality technology, and a camera controlling to obtain a captured image; detecting a reference point object by image processing the captured image; recognizing a code displayed on a reference point in the reference point object; obtaining 3D spatial model information matching the recognized code; setting 3D spatial information for the captured image based on the obtained 3D spatial model information; and displaying a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the 3D spatial information.

In this case, the obtaining of the three-dimensional space model information matching the recognized code may include obtaining the three-dimensional coordinates of the reference point object specifying the three-dimensional space of the captured image.

Also, the setting of the 3D spatial information for the captured image may include setting a 3D spatial coordinate system for the captured image based on the obtained 3D coordinates of the reference point object.

In addition, the step of obtaining 3D spatial model information matching the recognized code further includes obtaining a scale of the reference point object, and setting 3D spatial information for the captured image. may include obtaining the reference point scale information based on a size of the reference point object and a pixel displaying the reference point object.

In addition, the displaying of a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the three-dimensional spatial information may include determining the size of the virtual object matching the three-dimensional space of the captured image. The method may include augmenting the captured image according to reference point scale information.

In addition, the steps of updating the 3D spatial information by tracking changes in the position and orientation of the camera, and rendering and displaying the updated virtual object on the captured image according to the updated 3D spatial information. may include more.

In addition, the step of updating the 3D spatial information by tracking changes in the position and orientation of the camera may include updating the 3D spatial information based on real-time sensing data of an inertial sensor (IMU); The method may include correcting the updated 3D spatial information based on the position and direction of a reference point object recognized based on a visual inertial odometry (VIO) technology.

The method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention integrates data related to underground facilities and mixed reality (MR, Mixed Reality) technology in conjunction with a user's terminal to facilitate management and supervision of underground facilities Based on mixed reality technology, it is possible to provide users with 3D visualization of real-time information on underground facilities such as water and sewage, electricity, communication, gas pipelines, building electrical wiring, and heating and cooling piping, and to provide users with underground facilities and landfills. It has the effect of building a mixed reality service infrastructure that provides demonstration contents specialized for inspection and maintenance.

In addition, the method and system for providing a mixed reality related to an underground facility according to an embodiment of the present invention interact with the user's terminal in real time to update the underground facility information in real time according to a change in the location and/or orientation (attitude) of the user terminal by using mixed reality, there is an effect that it is possible to accurately and conveniently provide information on underground facilities that matches the space in which the user wants to check information on underground facilities.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention utilizes DGPS (Differential Global Positioning System, high-precision GPS) to match and match a three-dimensional space with an underground facility, thereby having improved accuracy. It has the effect of providing information on underground facilities.

In addition, the method and system for providing underground facility-related mixed reality according to an embodiment of the present invention implements image recognition/tracking/learning technology and 3D spatial recognition/camera tracking technology for providing information on underground facilities, thereby providing high quality underground It is possible to provide facility information, which has the effect of maximizing the efficiency of underground facility investigation, analysis, design, prevention of education, and supervision.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention utilizes a camera and/or an inertial sensor (IMU, Inertial Measurement Unit) of the terminal to determine the position and/or orientation (posture) of the user terminal. By providing information on underground facilities according to changes, there is an effect that it is possible to provide information on underground facilities according to changes in the state of the user terminal through its own algorithm and process without a separate location tracking device.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention is based on the 3D spatial information provided by the national spatial information platform (V-WORLD), the underground facilities buried underground on the electronic map drawing. By mapping, it is possible to provide a more realistic and accurate shape of underground facilities including numerical topography, and to ensure compatibility and performance of existing infrastructure and new software (S/W).

However, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned can be clearly understood from the description below.

1 is a conceptual diagram of a method and system for providing a mixed reality related to an underground facility according to an embodiment of the present invention.
2 is an internal block diagram of a terminal according to an embodiment of the present invention.
3 is an internal block diagram of an underground facility content providing server according to an embodiment of the present invention.
4 is a flowchart illustrating a method of constructing an underground facility database according to an embodiment of the present invention.
5 is a conceptual diagram for explaining a method for providing a mixed reality related to an underground facility based on GPS non-interworking according to an embodiment of the present invention.
6 is a flowchart illustrating a method for providing a mixed reality related to an underground facility based on GPS non-interworking according to an embodiment of the present invention.
7 is an example of a reference point according to an embodiment of the present invention, and is a diagram for explaining a method of automatically extracting an effective area from the reference point.
8 is an exemplary diagram for selecting an effective elliptical region through an Intersection Of Union (IOU) according to an embodiment of the present invention.
9 is a view for explaining a process of performing homography transformation in order to convert an elliptical reference point into a circular reference point according to an embodiment of the present invention.
10 is an example showing 3D spatial model information of a reference point according to an embodiment of the present invention.
11 is an example illustrating a state in which a rendered virtual object is displayed on a captured image according to an embodiment of the present invention.
12 is a flowchart for explaining a method for providing a mixed reality related to an underground facility based on GPS linkage according to another embodiment of the present invention.
13 is an example of obtaining 3D spatial information based on DGPS (High Precision GPS) according to another embodiment of the present invention.
14 is an example of tracking a position and orientation change according to the movement of a terminal based on a plurality of reference points based on a VIO (Visual Inertial Odometry) technology according to another embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them, will become apparent with reference to the embodiments described below in detail in conjunction with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first, second, etc. are used for the purpose of distinguishing one component from another, not in a limiting sense. Also, the singular expression includes the plural expression unless the context clearly dictates otherwise. In addition, terms such as include or have means that the features or components described in the specification are present, and do not preclude the possibility that one or more other features or components will be added. In addition, in the drawings, the size of the components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily indicated for convenience of description, the present invention is not necessarily limited to the illustrated bar.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when described with reference to the drawings, the same or corresponding components are given the same reference numerals, and the overlapping description thereof will be omitted. .

1 is a conceptual diagram of a method and system for providing a mixed reality related to an underground facility according to an embodiment of the present invention.

Referring to FIG. 1 , a method and system for providing a mixed reality related to an underground facility according to an embodiment of the present invention may include a terminal 100 and an underground facility content providing server 200 (hereinafter referred to as a content providing server).

Here, each component of FIG. 1 may be connected through a network. The network means a connection structure in which information can be exchanged between each node, such as the terminal 100 and the content providing server 200, and an example of such a network is a 3rd Generation Partnership Project (3GPP) network, a Long Term (LTE) network, and the like. Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth (Bluetooth) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, and the like are included, but are not limited thereto.

- terminal

First, in an embodiment of the present invention, the terminal 100 is a portable computing device possessed by a user who performs work at a site where underground facilities in each area are buried, and a program for performing mixed reality service related to underground facilities is provided. Installed portable terminals such as smart phones, digital broadcasting terminals, mobile phones, personal digital assistants (PDA), portable multimedia players (PMPs), navigation devices, tablet PCs, wearable devices, and smart glasses. may include. Here, the underground facility may mean a facility buried underground for water supply, sewage, electricity, communication, gas, oil transmission, heating, and the like.

The terminal 100 may provide information on underground facilities including various information related to underground facilities based on a code matched to a reference point 10 included in an image captured in a specific space in an embodiment of the present invention. .

For example, the reference point 10 may include at least one of a mark peg, a burial pin, a mark pole, a mark pin, a manhole, and a mark peg exposed to the ground while being linked with an underground facility. The role indicated by the reference point 10 may be marked with a first code, and for example, a first code indicating a water supply, sewerage, sewage pipe, street light line, underground transmission line or communication line is engraved on a metal sign nail. or embossed.

In addition, the reference point 10 may display a second code indicating the area where the reference point 10 is installed, for example, the second code indicating the area name, address name or road address name is engraved on a metal sign nail. or embossed.

In addition, a third code indicating the pipe diameter or pipe direction of the pipe connected to the reference point 10 may be displayed at the reference point 10, for example, an arrow indicating the pipe diameter value or the pipe direction is engraved or Can be embossed.

The reference point 10 serves as a reference index for recognizing the three-dimensional space of the captured image captured through the camera, and information about the reference point 10 and the underground facility matching the reference point 10 is obtained. It may be a mark containing the above codes (codes) that can.

In addition, the underground facility information here includes three-dimensional modeling images (virtual objects) of underground facilities, absolute location information, type information, size information, orientation information, underground analysis information (type of medium, depth of medium, etc.), completion information ( It may be information including at least one of completion drawings, completion location, completion date, etc.), management information (management history, etc.) and/or electronic drawings of underground facilities (virtual object set) information.

The terminal 100 may provide such information on underground facilities by 3D visualization based on mixed reality (MR) technology. In an embodiment, the terminal 100 may output and provide information about underground facilities as a 3D image on a captured image captured by a camera based on a mixed reality technology.

In detail, in the terminal 100 according to an embodiment of the present invention, in providing the underground facility information as a three-dimensional image, a background image (ie, of a real object) being photographed at a current location by a camera mounted on the terminal 100 . A mixed reality image may be provided by superimposing underground facility information (eg, a virtual object implemented as a 3D modeling image) on the set).

In this case, the virtual object according to the embodiment may refer to a virtual object that is stored in a database by matching the corresponding three-dimensional space without being seen in the three-dimensional space of the background image. For example, the virtual object may be a three-dimensional modeling image of an underground facility that is invisible as it is disposed underground in a corresponding three-dimensional space. Here, the 3D modeling image may be an image created by a mathematical model that can reproduce the shape of an underground facility in a virtual three-dimensional space, that is, the 3D modeling image is an image of the shape of the underground facility. It may be data in a form that a computer can understand, which is generated so that it can be checked in a dimensional space.

In addition, the real object according to the embodiment may mean an object that is actually seen on the 3D space of the background image, for example, a reference point including a sign nail, a manhole, etc. existing in the 3D space. (10) and the like.

In addition, the terminal 100 may display information related to an underground facility as a mixed reality image in which a real object that is a real physical object and a virtual object that is a virtual object generated based on information on the underground facility can interact.

For example, when the underground facility is a pipeline, the terminal 100 may display information on a pipeline provided in the underground space of the current location in the form of a three-dimensional network map or augmented reality view. That is, the user may be provided with information on the underground facilities buried based on his/her location through the terminal 100 , and may receive a 3D modeling image whose viewpoint is changed according to the shooting direction of the camera.

For example, when the terminal 100 photographs the ground, information on the underground facilities of the underground facilities matching the ground currently illuminated by the user in consideration of the current location of the terminal 100 may be displayed on the display. In addition, when the user selects specific coordinates of the electronic map displayed on the terminal 100, the terminal 100 displays a virtual underground space based on the coordinates and information on underground facilities located in the corresponding underground space. may also be displayed.

In addition, the terminal 100 can obtain identification information by recognizing a code (mark) for an underground facility on the site and transmit it to the content providing server 200 through the information and communication network, and three-dimensional matching the identification information The spatial model information can be received and used.

Here, the three-dimensional space model information is information stored in a memory and/or database by matching with each reference point 10, and three-dimensional coordinates (ie, absolute position) of the reference point 10 specifying a three-dimensional space, a reference point It may include information on underground facilities matching the scale and/or reference point 10 of (10).

The terminal 100 according to an embodiment may capture the code to obtain identification information and transmit it to the content management server. At this time, the content management server may provide the terminal 100 by reading the three-dimensional space model information matching the received identification information from the database, through which the terminal 100 acquires information about underground facilities and can make use of it.

Here, in order to implement the above-described function, the terminal 100 may be loaded with a recording medium in which an application program for interworking with a content management server through a network, displaying the received information and reproducing a 3D modeling image is recorded.

In the above, the terminal 100 obtains identification information by recognizing the code of the reference point 10 matched with the underground facility, and transmits the obtained identification information to the content providing server 200 to receive 3D spatial model information Although it has been described that the underground facility information is acquired based on this, various embodiments are also possible, such as the terminal 100 being able to perform a series of operations of the content providing server 200 for acquiring the underground facility information by itself. .

In addition, in the embodiment, the terminal 100 may acquire updated underground facility information in real time according to a change in the position and/or orientation (attitude) of the terminal 100, and based on the obtained information, the user's terminal Underground facility information optimized for the (100) state (ie, the location and/or orientation of the terminal 100) may be provided to the user through the mixed reality.

In this way, according to the above-described structure, the terminal 100 according to an embodiment of the present invention provides the user with information on underground facilities matching a specific space as a three-dimensional image according to the user's terminal 100 state, so that the user It is possible to easily check the buried state of underground facilities.

Hereinafter, the structure of the terminal 100 according to an embodiment of the present invention will be described in detail with reference to the drawings. 2 is an internal block diagram of the terminal 100 according to an embodiment of the present invention.

Referring to FIG. 2 , in an embodiment of the present invention, the terminal 100 includes a communication unit 110 , a display unit 120 , an interface unit 130 , an inertial sensor (IMU, Inertial Measurement Unit, 140 ), and a camera 150 . ), a memory 160 , and a processor 170 .

First, the communication unit 110 may transmit and receive various data and/or information for providing a mixed reality service related to an underground facility.

In an embodiment, the communication unit 110 may connect to an information communication network and communicate with the content providing server 200 to transmit/receive data (eg, underground facility information, etc.) related to an underground facility-related mixed reality service.

This communication unit 110, the technical standards or communication methods for mobile communication (eg, Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), High Speed Downlink Packet Access (HSDPA), HSUPA ( High Speed Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.) Signals can be sent and received.

Next, the display unit 120 may output various information related to the mixed reality service related to the underground facility as a graphic image.

In an embodiment, the display unit 120 may display a mixed reality image configured by the processor 170 of the terminal 100 including a graphic user interface (GUI).

The display unit 120 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. (a flexible display), a three-dimensional display (3D display), and may include at least one of an e-ink display (e-ink display).

Next, the interface unit 130 may receive an input of a user's operation related to the underground facility-related mixed reality providing service.

The interface unit 130 is configured integrally with the display unit 120 as well as a key button provided in the terminal 100 and is a touch screen that clicks a virtual button displayed on the screen. can be implemented in the form of

Next, the inertial sensor IMU 140 may be implemented including an acceleration sensor and/or a gyro sensor, and may measure a change in the position and/or orientation of the terminal 100 .

In an embodiment, the inertial sensor (IMU, 140) transmits the current azimuth data to the processor 170 when the terminal 100 displays the mixed reality image so that a three-dimensional modeling image (virtual object) corresponding to the corresponding viewpoint is generated. can be output.

Next, the camera 150 may be disposed on the front and/or rear of the terminal 100 to acquire an image by photographing the disposed direction side.

In an embodiment, the camera 150 may photograph a space (area) in which the user desires to obtain information on the underground facility and/or the code of the reference point 10 matched with the underground facility.

In particular, when the camera 150 according to the embodiment displays the mixed reality image based on the captured image for the surrounding space of the area where the terminal 100 is located, the underground facility information is superimposed on the captured image. have.

Next, the memory 160 may store any one or more of an operating system (OS), various application programs, data, and commands for providing a mixed reality service related to an underground facility.

The memory 160 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and may be a web storage that performs a storage function of the memory 160 on the Internet. may be In addition, the memory 160 may be a recording medium in a removable form in the terminal 100 .

Next, the processor 170 may control the overall operation of each of the above-described components in order to implement the underground facility-related mixed reality providing service.

That is, the processor 170 may be a system-on-chip (SOC) suitable for the portable terminal 100 including a central processing unit (CPU) and a graphic processing unit (GPU), and an operating system (SOC) stored in the memory 160 . OS) and application programs, and can control each component mounted on the terminal 100 .

The processor 170 may communicate with each component internally by a system bus, and may include one or more predetermined bus structures including a local bus.

In addition, these processors 170, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers (controllers) ), micro-controllers, microprocessors, and other electrical units for performing other functions.

- Server for providing content for underground facilities

Meanwhile, in an embodiment of the present invention, the content providing server 200 may provide various types of information generated on the underground facility database based on spatial information related to a plurality of underground facilities.

In detail, the content providing server 200 stores and manages various information related to underground facilities as a database based on the spatial information related to underground facilities obtained from the national spatial information platform (V-WORLD) and/or the terminal 100 . and transmit the stored information to the terminal 100 to provide it to the user. A detailed description thereof will be provided later.

Here, the above-described spatial information related to underground facilities may be a computer file provided by a national spatial information portal or a department operated by the Ministry of Land, Infrastructure and Transport, or a computer file based on an image obtained through the camera 150 of the terminal 100. may be

3 is an internal block diagram of the underground facility content providing server 200 according to an embodiment of the present invention. In detail, referring to FIG. 3 , the content providing server 200 in the embodiment of the present invention may include a data receiving unit 210 , a data transmitting unit 220 , a data processing unit 230 , and a database 240 .

First, the data receiving unit 210 and the data transmitting unit 220 may exchange various data for performing mixed reality service related to the underground facility with the terminal 100 and/or an external server through a network.

Next, the data processing unit 230 may perform a series of data processing for performing a mixed reality service related to an underground facility.

In this case, the data processing unit 230 may include a DB (database) building unit 231 , a three-dimensional space sensing unit 232 , and an information arrangement unit 233 .

In detail, first, the DB building unit 231 may generate information for building an underground facility database based on the underground facility information, and may store the generated information into a database and store it on the database 240 .

In detail, the DB construction unit 231 includes an information input unit that receives spatial information related to underground facilities transmitted through an information communication network or stored in a recording medium, and a three-dimensional modeling object (virtual object) corresponding to a plurality of underground facilities. A 3D rendering unit, a spatial information mapping unit that maps a 3D modeling object in a 3D virtual space defined by spatial information, and a reference point 10 providing unit that manages information related to a reference point 10 matching an underground facility may include

At this time, the above-described reference point 10 providing unit, the three-dimensional coordinates (absolute position) for each reference point 10, the size (scale) of the reference point 10, and the reference point 10 and matching underground facility information, etc. It is possible to generate and provide 3D spatial model information based on this.

In addition, the 3D space sensing unit 232 may determine a 3D space matching the captured image based on the captured image obtained from the terminal 100 and the spatial information related to the underground facility stored in advance in the database 240 . have.

In addition, the 3D space sensing unit 232 may read a virtual object matching the 3D space from the database 240 and generate scene specification information corresponding to the 3D space based on this.

Here, the scene specification information is information that detects a virtual object matching the three-dimensional space photographed by the camera 150 of the terminal 100, and assists in matching the detected virtual object with the corresponding three-dimensional space. A detailed description will be given later.

In addition, the information arrangement unit 233 renders each virtual object to be placed at an appropriate location in the 3D space based on the scene specification information and underground facility information generated through the 3D space sensing unit 232 . rendering) can be performed.

In addition, the data processing unit 230 including the above configuration unit, according to the change of the real-time position and / or orientation according to the change in the movement of the terminal 100 in the embodiment of the present invention, scene specification information and the rendered virtual object It can be updated in real time.

The data processing unit 230 is, ASICs (application specific integrated circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), controllers (controllers) , micro-controllers, microprocessors, and other electrical units for performing other functions.

Finally, the database 240 may store various data related to the underground facility-related mixed reality service.

In an embodiment, in the database 240 , an underground facility database including virtual objects related to a plurality of underground facilities, 3D spatial model information and/or underground facility information, etc. may be built, stored, and managed.

The database 240 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and is a web storage that performs a storage function of the database 240 on the Internet. may be

In more detail, in the embodiment of the present invention, the database 240 may match and store the three-dimensional coordinates of the reference point 10 included in the three-dimensional space with the corresponding three-dimensional space.

In addition, the database 240 may match and store the absolute coordinates of the virtual object (ie, the 3D modeling image of the underground facility) matched in the 3D space with the 3D space.

In addition, the database 240 is an electronic drawing that is a map generated by mapping underground facilities buried underground and a three-dimensional space based on spatial information related to underground facilities provided by the national spatial information platform (V-WORLD). can be saved.

Here, the electronic drawing may be information implemented as a three-dimensional graphic map of underground facilities buried underground, and may be a set of virtual objects in an embodiment.

And the database 240 may include an underground facility database including a plurality of information as described above.

Hereinafter, an embodiment of constructing an underground facility database in the database 240 will be described in detail with reference to the accompanying drawings.

4 is a flowchart illustrating a method of constructing an underground facility database according to an embodiment of the present invention.

Referring to FIG. 4 , first, the content providing server 200 may acquire underground facility data based on an underground facility network map owned by a local government (hereinafter referred to as a local government). (S101)

Here, the underground facility data is information included in the underground facility information, and may include absolute location, type, size, and/or orientation information of the underground facility.

In detail, the content providing server 200 performs communication with an underground facility-related server and/or a national spatial information platform server operated by a local government, and obtains underground facility data including a network map of an underground facility owned by the local government. And based on this, information on underground facilities can be built.

For example, the content providing server 200 may construct an electronic drawing of an underground facility by receiving a water/sewage pipe network map from an underground facility-related server operated by a local government.

Also, the content providing server 200 may acquire underground facility data from a management subject of the underground facility. (S103)

In detail, the content providing server 200 communicates with a server of a specialized institution that manages underground facilities, such as Korea Electric Power Corporation and Korea Gas Corporation, to obtain drawings or documents related to underground facilities, and based on this, information on underground facilities can be built

In addition, the content providing server 200 may acquire underground facility data by measuring the status of codes for each underground facility disposed on the site. (S105)

In detail, the content providing server 200 may generate information about underground facilities by identifying the status of codes assigned to reference points 10 that are matched to each of the underground facilities and installed in the field.

For example, the content providing server 200, based on the identification information and/or 3D spatial model information stored in the database 240 in response to the code given to the reference point 10 matching each underground facility, Information such as the number of sign nails and/or manholes installed at the site, the absolute location and/or type of underground facilities matching the sign nails and/or manholes, etc. can be obtained, and information on underground facilities can be built based on this have.

That is, the content providing server 200 may build underground facility information based on the underground facility data obtained in various ways as described above, and may store the constructed underground facility information in the database 240 . (S107)

In detail, the content providing server 200 may generate a 3D modeling image (virtual object) for each underground facility, 3D spatial model information and/or underground facility information, etc. based on the acquired underground facility data, and based on this to build an underground facility database.

In addition, the content providing server 200 may perform underground radar exploration (GPR, Ground Penetrating Radar) to additionally collect underground facility data, and store it in the database 240 . (S109)

In detail, the method and system for providing a mixed reality related to an underground facility may further include an underground facility surveying device according to an embodiment.

Here, the underground facility surveying device may acquire underground facility data by measuring the underground facility using the cadastral reference point 10 or by exploring the underground facility through electromagnetic induction. In addition, the underground facility surveying apparatus may transmit the obtained underground facility data to the content providing server 200 through the network.

In more detail, the content providing server 200 may receive the underground facility data obtained from the underground facility surveying device.

At this time, in an embodiment, the underground facility surveying apparatus may detect a code given to the reference point 10 existing in the three-dimensional space, and based on the sensed code, the three-dimensional space matched with the reference point 10 It is possible to obtain model information and the like.

In addition, in an embodiment, the apparatus for surveying underground facilities may acquire absolute position, orientation and/or size information of underground facilities buried underground based on electromagnetic guided exploration. In addition, in an embodiment, the underground facility surveying apparatus may obtain absolute position and/or orientation information of the underground facility for the space in which the exploration is performed in conjunction with the GPS.

Thereafter, the content providing server 200 may store the additionally collected underground facility data in the database 240 as described above, and update the underground facility information based on the additionally collected underground facility data.

In addition, the content providing server 200 may build an underground facility database based on the obtained underground facility information. (S111)

In detail, the content providing server 200 may build an underground facility database including a virtual object category, a 3D spatial model information category, and/or an underground facility information category.

In more detail, the underground facility database may store a virtual object in which the underground facility is modeled in three dimensions in the virtual object category.

In addition, the underground facility database is the three-dimensional coordinates (ie, absolute position) of the reference point 10 that specifies the three-dimensional space matched with the reference point 10 in the three-dimensional space model information category (ie, absolute position), the size of the reference point 10 ( scale) and/or 3D spatial model information including information on underground facilities matching the reference point 10 and the like may be stored.

In addition, the underground facility database includes three-dimensional modeling images (virtual objects), absolute location information, type information, size information, orientation information, underground analysis information (type of medium, depth of medium, etc.) of underground facilities in the underground facility information category. ), completion information (completion drawing, construction location, completion date, etc.), management information (management history, etc.), and/or underground facility information including at least one of underground facility electronic drawing (virtual object set) information can be stored. .

As such, the content providing server 200 can systematically manage the data that is the basis of the underground facility related mixed reality service by building the underground facility database for the underground facility related mixed reality providing service as described above. It is possible to improve the quality of the mixed reality service related to underground facilities by providing high-quality data.

In the above, the content providing server 200 builds an underground facility database, detects a three-dimensional space to generate scene specification information, and renders a virtual object based on the generated scene specification information and underground facility information, etc. Although it has been described that a series of operations of , the terminal 100 may perform a part of the operations performed by the content providing server 200, various embodiments may also be possible.

- Method of providing mixed reality related to underground facilities based on non-GPS connection

Hereinafter, with reference to the accompanying drawings, a method for providing a mixed reality related to an underground facility based on a non-interlocking GPS system will be described in detail. In more detail, a method of recognizing the reference point 10 without interworking with GPS to acquire an initial orientation (posture) between the space-camera 150 and providing a mixed reality related to underground facilities based on this will be described in detail.

5 is a conceptual diagram for explaining a method for providing a mixed reality related to an underground facility based on a GPS non-interlocking according to an embodiment of the present invention, and FIG. 6 is a GPS non-interworking based mixed reality related to an underground facility according to an embodiment of the present invention. It is a flowchart to explain the method.

5 and 6 , first, the terminal 100 may acquire a captured image by controlling the camera 150 . (S201)

In detail, the terminal 100 may acquire a photographed image captured by the camera 150 of a space (area) in which the user wants to acquire information about underground facilities.

In addition, the terminal 100 may image-process the acquired captured image to detect an object of the reference point 10 . (S203)

In detail, the terminal 100 first detects the object to be identified in the captured image, identifies the real object, matches each underground facility among the identified real objects, and is installed in the field, thereby specifying each underground facility reference point 10 ) object can be detected.

Here, the reference point 10 may be a reference index on the ground for recognizing the three-dimensional space of the captured image, and at least the reference point 10 and information on the underground facility matching the reference point 10 can be obtained. It may include one or more codes.

For example, referring to FIG. 7 , the reference point 10 may be a mark nail installed in the field, and may be implemented by including at least one or more codes (c1, c2, c3, ...).

For example, the reference point 10 may include at least one of a mark peg, a burial pin, a mark pole, a mark pin, a manhole, and a mark peg exposed to the ground while being linked with an underground facility. The role indicated by the reference point 10 may be indicated by a first code c1, for example, a first code c1 indicating water supply, sewerage, sewage pipe, street light line, underground transmission line or communication line, etc. It may be marked with a engraving or weak engraving on the sign nail made of metal.

In addition, the second code c2 indicating the area where the reference point 10 is installed may be displayed on the reference point 10 , and for example, the second code c2 indicating the area name, address name, or road address name may be displayed on the reference point 10 . It may be marked with engraving or embossing on the sign nail, which is a metal material.

In addition, a third code c3 indicating the pipe diameter or pipe direction of the pipe connected to the reference point 10 may be displayed at the reference point 10, for example, a third code including an arrow indicating a pipe diameter value or a pipe direction. 3 The code (c3) may be marked in engraving or embossing on a metal sign.

At this time, the terminal 100 is to interwork with a deep-learning neural network installed in the terminal 100, the content providing server 200, and/or an external server to perform image deep learning. can Here, the deep learning neural network is at least one of a convolutional neural network (CNN), Regions with CNN features (R-CNN), Fast R-CNN, Faster R-CNN, and Mask R-CNN. It may include a neural network.

Next, the terminal 100 may detect an ellipse from the detected reference point 10 object. (S205) In detail, the reference point 10 is a circular sign on the road such as a sign nail, and may be recognized as an oval-shaped object in the photographed image by the photographing direction.

Accordingly, the terminal 100 may first perform an ellipse detection algorithm using “Projective Invariant Pruning” in order to more clearly recognize the detected reference point 10 .

In detail, the circular reference point 10 is often recognized as an ellipse depending on the direction in which it is photographed. However, when the reference point 10 is recognized as an ellipse, it may be difficult to accurately detect a code assigned to the reference point 10 . Therefore, in the embodiment of the present invention, the code of the reference point 10 is accurately detected by applying the ellipse detection algorithm using "Projective Invariant Pruning", and the overall shape even if the edge of the ellipse of the reference point 10 is partially covered. can be detected.

7 is an example of a reference point 10 according to an embodiment of the present invention, and is a diagram for explaining a method of automatically extracting an effective area from the reference point 10 .

In more detail, referring to FIG. 7 , the terminal 100 may first detect an ellipse from the reference point 10 .

In this case, the terminal 100 may perform a processing algorithm for removing false positives among the detected ellipses. Here, a false positive is defined as an error of judging that there is something that does not exist by selecting the wrong ellipse.

That is, the circular reference point 10 has a product with a border. In this case, the terminal 100 recognizes arcs in different regions in ellipse detection to remove unwanted ellipse detection. processing algorithm can be applied.

8 is an exemplary diagram for selecting an effective elliptical region through an Intersection Of Union (IOU) according to an embodiment of the present invention.

Referring to FIG. 8 , the ellipse recognition rate score is proportional to the overlapping Arc, and it is necessary to remove the overlapping ellipse on the high-score ellipse. The method of automatically selecting the elliptical area was applied. For reference, an ellipse having an Intersection Of Union (IOU) score greater than a predetermined set value may be selected, or an ellipse may be selected through a relative comparison.

That is, the terminal 100 recognizes at least one ellipse region in automatically extracting a region in which a code is displayed from the reference point 10 of the photographed image, and recognizes at least one ellipse region and an effective ellipse through Intersection Of Union (IOU). The area can be automatically selected.

9 is a view for explaining a process of performing homography transformation in order to convert an elliptical reference point 10 into a circular reference point 10 according to an embodiment of the present invention.

In addition, referring to FIG. 9 , the terminal 100 automatically selects an effective elliptic region through Intersection Of Union (IOU) based on at least one or more elliptic regions, and then performs a homography in order to remove geometric distortion of the ellipse. ) by performing the transformation, it is possible to obtain a circular-shaped reference point 10 object and homography transformation information. (S207)

That is, the terminal 100 may transform the elliptical reference point 10 image into a circular reference point 10 image by performing homography transformation on the reference point 10 , through which the reference point 10 image is transformed. ), by acquiring an image in the form of viewing the object vertically from the top, it is possible to more accurately grasp the code and/or size of the corresponding reference point 10 .

In this case, the terminal 100 may obtain homography conversion information through a homography conversion process.

Here, the homography transformation information includes change element information (eg, gradient change information of a circle, size change information, horizontal diameter change information or vertical diameter change information, etc.). Such homography conversion information may be used as base data for acquiring an initial camera posture of the terminal 100 later.

Also, the terminal 100 may recognize a code assigned to the reference point 10 by deep learning the circular reference point 10 object. (S209)

In detail, the terminal 100 performs image deep learning (Identification) based on the circular reference point 10 to specify the reference point 10 and the underground facility matching the reference point 10 (code) can be detected.

At this time, in the embodiment, the terminal 100 may perform image deep learning for code detection through a convolutional neural network (CNN), and through the convolutional neural network, The code assigned to the reference point 10 may be detected by extracting the feature map and deriving the code assigned to the reference point 10 based on the extracted feature map.

At this time, the terminal 100 may detect at least one or more plurality of codes through image deep learning for the reference point 10 object, and based on the detected at least one or more codes, the reference point 10 is matched to the corresponding reference point 10 later. Accurately obtain 3D spatial model information.

Subsequently, the terminal 100 recognizing the code based on the circular reference point 10 object may obtain 3D spatial model information matching the recognized code. (S211)

In detail, the terminal 100 may obtain 3D spatial model information matching the recognized code by searching the database 240 . Here, in the present embodiment, it has been described that the terminal 100 obtains 3D spatial model information by searching the database 240 , but it is also possible to obtain 3D spatial model information matching the code recognized in the memory 160 . Various embodiments are possible. However, it will be the most preferred embodiment to obtain 3D spatial model information from the database 240 more suitable for managing a large amount of data.

That is, the terminal 100 obtains 3D spatial model information matching the recognized code, and 3D coordinates for the corresponding reference point 10 (eg, coordinates indicating the absolute position of the reference point 10), the reference point The scale of (10), information on underground facilities matching the reference point (10), and the like can be obtained.

In this case, the size of the reference point 10 may be information further including the number of pixels indicated by the reference point 10 .

For example, referring to FIG. 10 , the terminal can recognize the code given to the corresponding sign by deep learning the image of the circular-shaped sign nail object, and the database 240 based on the identification information for the recognized code. 3D spatial model information matching the corresponding sign can be obtained by searching for .

Next, the terminal 100 may set 3D spatial information based on the obtained homography transformation information and 3D spatial model information. (S213)

Here, the 3D spatial information is information that provides position and orientation related information based on the viewpoint of the terminal looking in 3D space, and is matched to the initial camera posture, 3D spatial coordinate system, initial camera position, and 3D spatial coordinate system. Underground facility information and/or reference point scale information may be included.

First, the terminal 100 may acquire an initial camera posture based on the obtained homography transformation information and the three-dimensional coordinates of the reference point 10 .

Here, the initial camera posture may be information indicating the camera 150 posture (ie, orientation) when the camera 150 of the terminal 100 starts shooting (initial) the three-dimensional space including the reference point 10. have.

That is, the initial camera posture is information obtained by acquiring coordinate values for pitch, yaw, and roll coordinates, which are direction tracking coordinate systems, based on the three-dimensional coordinates of the reference point 10 in three-dimensional space. It may be azimuth information determined in which direction the camera 150 of the terminal 100 is facing.

Here, the pitch coordinate system among the direction tracking coordinate systems in the three-dimensional space can sense the x-axis rotation of the terminal 100, that is, the vertical rotation, and the yaw coordinate system can detect the y-axis rotation of the terminal 100, that is, the left-right rotation. In addition, the roll coordinate system may detect the z-axis rotation of the terminal 100 , that is, the forward and backward rotation.

Returning again, the terminal 100 in detail, the reference point 10 in the process of being transformed from an ellipse to a circular shape, the reference point 10, the change element information generated in the object (eg, the inclination change information of the circle, the horizontal diameter) Initial posture by the camera 150 of the terminal 100 based on the homography transformation information meaning change information, vertical diameter change information, etc.) and the three-dimensional coordinates of the reference point 10 included in the three-dimensional spatial model information (azimuth) can be estimated, and the initial camera posture can be obtained through this.

In addition, the terminal 100 may set a three-dimensional space coordinate system capable of tracking the position of the terminal in a three-dimensional space based on the three-dimensional coordinates of the reference point 10 of the obtained three-dimensional space model information.

Here, the three-dimensional space coordinate system may be generated based on the reference point 10 by matching one by one for one three-dimensional space, and the three-dimensional coordinates of the reference point 10 implemented as the x-axis, y-axis, and z-axis. It may be generated as an extension line of , or may be generated by being designated at a specific location on the 3D space based on the 3D coordinates of the reference point 10 according to a predetermined method.

Also, the terminal 100 may obtain information about underground facilities matching the 3D spatial coordinate system based on the set 3D spatial coordinate system and the underground facility information matching the reference point 10 of the 3D spatial model information.

In detail, the terminal 100 searches the memory 160 and/or the database 240 based on the underground facility information matching the reference point 10 and the 3D spatial coordinate system, and the underground facility information matched to the 3D spatial coordinate system can be obtained.

In addition, the terminal 100 may obtain information on underground facilities matching the three-dimensional spatial coordinate system and use it to later match virtual objects representing underground facilities in the three-dimensional space.

In addition, the terminal 100 obtains reference point scale information based on the size of the reference point 10 of the 3D spatial model information obtained based on the reference point 10 and pixel information indicating the reference point 10 object. can

Here, the reference point scale information is displayed on the captured image based on the preset size of the reference point 10 (eg, the diameter or radius of the circular reference point 10 ) and the pixel for displaying the size of the reference point 10 . It may be information for determining the scale of the reference point 10 to be used.

That is, the terminal 100 may obtain the above reference point scale information and perform a process of adjusting the scale of the virtual object according to the reference point 10 scale later.

Also, the terminal 100 may set the initial camera position based on the obtained 3D spatial coordinate system and reference point scale information.

In detail, the terminal 100 may first acquire distance information between the reference point 10 and the terminal according to the size of the reference point 10 based on the reference point scale information.

Here, the reference point 10-terminal distance information may be information obtained by calculating a physical distance between the reference point 10 captured by the camera 150 of the terminal 100 and the corresponding terminal 100 .

In detail, the terminal 100 acquires information on the scale, that is, the size (eg, the diameter or radius of the circular reference point 10 ) of the reference point 10 in the image taken with the camera 150 based on the reference point scale information In addition, according to the scale of the obtained reference point 10 , the distance between the terminal 100 photographing the corresponding reference point 10 and the reference point 10 may be calculated.

At this time, the distance between the reference point 10 and the terminal 100 according to the reference point 10 scale in the captured image is a predetermined reference (eg, the distance from the preset terminal 100 according to the diameter of the reference point 10, etc.) may be preset, and may be pre-stored in the memory 160 of the terminal 100 and/or the database 240 of the content providing server 200 .

For example, if the diameter of the reference point 10 of the captured image on the memory 160 is 1 cm or more and less than -3 cm, the reference point 10 - the terminal 100 distance is 50 cm, 3 cm or more and less than -5 cm. In this case, if the reference point 10-terminal 100 distance is 30 cm, 5 cm or more and less than -7 cm, the reference point 10-terminal 100 distance may be pre-stored as 10 cm. And when it is determined that the diameter of the reference point 10 of the captured image is 5 cm through the reference point scale information, the terminal 100 may determine the distance between the reference point 10 and the terminal 100 as 10 cm, and the reference point ( 10) - The terminal distance information can be determined to be 10 cm.

In addition, the terminal 100 may determine the position of the terminal 100 based on the three-dimensional spatial coordinate system based on the obtained reference point 10-terminal distance information, and Based on the initial camera position can be set.

As described above, in the embodiment of the present invention, the terminal 100 may convert 3D spatial information set based on the reference point 10 into 3D spatial information based on the terminal 100 as described above.

For example, the terminal 100 may obtain a 3D spatial coordinate system set based on the location of the terminal 100 and/or underground facility information matching the 3D spatial coordinate system and use it as 3D spatial information. have.

That is, the terminal 100 may acquire 3D spatial information in various ways to implement the underground facility-related mixed reality providing service according to the embodiment of the present invention.

To this end, the terminal 100 that has set the 3D spatial information as described above may acquire scene specification information based on the set 3D spatial information. (S215)

Here, the scene specification information is information that detects a virtual object matching the three-dimensional space photographed by the camera 150 of the terminal 100 and assists in matching the detected virtual object with the corresponding three-dimensional space. It may include at least one of object position (coordinate) setting information, virtual object orientation setting information, and virtual object scale setting information.

In detail, the terminal 100 is based on the 3D spatial coordinate system of the acquired 3D spatial information and the underground facility information matching the 3D spatial coordinate system, the position of the virtual object acquired in the 3D space of the captured image ( coordinates) can be set, and virtual object location (coordinates) setting information can be created.

For example, the terminal 100 may obtain the position of the virtual object in the 3D space of the captured image based on the 3D spatial coordinate system and the absolute position information of the underground facility information matching the 3D spatial coordinate system. and, through this, virtual object location (coordinate) setting information can be generated.

In addition, the terminal 100 may set the orientation of the virtual object obtained in the three-dimensional space of the captured image based on the initial camera posture of the obtained three-dimensional spatial information, and may generate virtual object orientation setting information. .

That is, the terminal 100, based on the orientation of the terminal 100 obtained through the initial camera posture, can determine the direction (orientation) of the viewpoint that the terminal 100 has in the three-dimensional space, The virtual object orientation setting information may be generated so that the viewpoint direction of the terminal 100 matches the orientation of the virtual object.

In addition, the terminal 100 may set the scale of the virtual object obtained for the three-dimensional space of the captured image based on the reference point scale information of the obtained three-dimensional spatial information, and generate virtual object scale setting information can do.

Here, the virtual object scale setting information may be information in which the size of the virtual object is set to be displayed on the three-dimensional space of the captured image in proportion to the size of the reference point 10 .

In detail, the terminal 100 first, a pixel for displaying the size of the reference point 10 (eg, the diameter or radius of the circular reference point 10 ) and the size of the reference point 10 preset through reference point scale information information on the number of .

In addition, the terminal 100 may detect a pixel on the same line as the position of the reference point 10 in the three-dimensional space, that is, a plurality of pixels indicating the same distance as the reference point 10 .

At this time, when the terminal 100 displays the virtual object based on a plurality of pixels indicating the same distance as the detected reference point 10 , the size of the reference point 10 is proportional to the size of the reference point 10 and the number of pixels. The virtual object scale information can be generated so that the corresponding virtual object is displayed as the number.

For example, the terminal 100 may estimate the number of pixels according to the size of the reference point 10 in units of meters (m), and may determine the number of pixels according to the size of the virtual object based on this.

In detail, for example, the terminal 100 may be preset to estimate 0.01 meter as 1 pixel, and it may be confirmed that the size of the reference point 10 is 0.1 meter based on the preset size information of the reference point 10 . have. In addition, the terminal 100 may determine that the reference point 10 is output based on 10 pixels in a 3D space according to a preset metric unit estimation principle.

That is, the terminal 100 may obtain reference point scale information indicating that the reference point 10 is displayed based on 10 pixels in an image outputting a 3D space when the size of the reference point 10 is 0.1 meter.

At this time, when the terminal 100 displays the virtual object based on a plurality of pixels indicating the same distance as the reference point 10 , the preset size of the virtual object based on the underground facility information for the virtual object It can be confirmed that, for example, is 1 meter.

And the terminal 100 sets the number of pixels according to the size of the obtained reference point 10 to 100 pixels in proportion to the information that the number of pixels according to the size of the reference point 10 is 10 pixels, that is, when the size of the corresponding virtual object is 1 meter. can be determined, and virtual object scale information can be generated based on this.

That is, the terminal 100 provides the virtual object scale information (virtual object size: 1 meter, the number of display pixels: 100 pixels) proportional to the reference point scale information (the size of the reference point 10: 0.1 meter, the number of display pixels: 10 pixels). ) to display the virtual object in a scale that matches the background image in 3D space.

In addition, the terminal 100 sets the virtual object scale for a plurality of pixels displaying the same distance as the corresponding reference point 10 based on the reference point scale information, and then for the remaining plurality of pixels displaying the corresponding virtual object. , the number of pixels according to the size of the virtual object may be determined according to a predetermined degree of perspective based on the virtual object scale set for pixels on the same distance as the reference point 10 .

For example, when the virtual object scale information for a plurality of pixels displaying the same distance as the reference point 10 is set to 'virtual object size: 1 meter, number of display pixels: 100 pixels', the terminal 100 For a plurality of pixels that display a virtual object for a distance rather than a distance, the number of display pixels can be set to gradually decrease to less than 100 pixels as the distance increases. As the distance increases, the number of display pixels may be set to gradually increase to exceed 100 pixels.

That is, the terminal 100 determines the scale of the virtual object based on the scale information of the reference point 10 to correspond to real objects (in the embodiment, the reference point 10) that exist in the three-dimensional space of the captured image. The virtual object can be augmented and displayed to the desired size.

Next, the terminal 100 having obtained the scene specification information may render a virtual object based on the obtained scene specification information and display it as a mixed reality (MR) image. (S217)

11 is an example illustrating a state in which a rendered virtual object is displayed on a captured image according to an embodiment of the present invention.

In detail, referring to FIG. 11 , the terminal 100 matches virtual objects matching on the three-dimensional space of the captured image based on the obtained scene specification information, the initial camera posture and the initial camera position of the three-dimensional spatial information. It can be rendered.

In more detail, the terminal 100 may determine the camera 150 viewpoint of the terminal 100 based on the obtained initial camera posture and initial camera position, that is, the posture (azimuth) and position of the terminal, and scene specification information Rendering may be performed by matching virtual objects matching the 3D space of the captured image according to the camera 150 viewpoint determined based on the .

In this case, the terminal 100 may display the virtual object matched and rendered in the three-dimensional space overlaid on the image captured by the camera 150 through the display unit 120 .

In detail, in order to superimpose the virtual object on the three-dimensional space of the captured image in an accurate position, the terminal 100 first determines the coordinates previously assigned to each real object existing in the three-dimensional space based on the three-dimensional space coordinate system. It may be derived from memory 160 and/or database 240 .

In addition, the terminal 100 may display a real object positioned in a three-dimensional space based on the derived coordinates for each real object, and each virtual object to be matched in the three-dimensional space, by matching and overlapping the display.

In addition, the terminal 100 may give a predetermined transparency to the virtual object overlapped and displayed in the 3D space so that the virtual object can be displayed naturally on the captured image.

In addition, the terminal 100 may match and store the positional relationship between the rendered virtual object and the real object of the captured image in three dimensions as described above, and through this, the photographing time by the camera 150 of the terminal 100 is Even if it changes, the virtual object can be moved according to the location of the real object according to the change of the shooting point, so that it can be accurately matched and displayed on the real object.

In detail, the terminal 100 may update the scene specification information by tracking a change in the position and orientation (posture) of the camera 150 of the terminal 100, and a captured image based on the updated scene specification information Augmented rendering may be performed to match virtual objects matching the three-dimensional space of , and the rendered virtual objects may be displayed by superimposing them on the captured image. (S219)

In more detail, the terminal 100 may track a change in the position and orientation of the camera 150 of the terminal 100 based on the inertial sensor (IMU, 140), and the camera of the terminal 100 obtained through tracking ( 150) Based on the change in position and orientation, it is possible to update the scene specification information for the 3D space that matches the change.

In addition, the terminal 100 renders a virtual object matching the three-dimensional space of the captured image based on the updated scene specification information, and displays it overlaid on the captured image of the camera 150 through the display unit 120 to be output. have.

At this time, the terminal 100 matches the positional relationship between the augmented-rendered virtual object and the real object of the captured image based on the initial camera posture and the initial camera position three-dimensionally and based on the stored information, the terminal 100 ), the virtual object can be moved to match the position of the real object according to the change in the shooting time when the shooting time is changed by the camera 150 of Objects can be accurately and conveniently registered and displayed in 3D space.

That is, the terminal 100 may display the virtual object superimposed on the real object by matching the positions of the real object and the virtual object in the three-dimensional space, and in this case, by giving a predetermined transparency to the virtual object, the user The real object behind the virtual object can be recognized together, and even if the shooting time by the camera 150 of the terminal 100 changes, the virtual object can be moved to match the changed location of the real object and accurately overlapped and displayed. .

Meanwhile, FIG. 12 is a flowchart illustrating a method for providing mixed reality related to an underground facility based on GPS linkage according to another embodiment of the present invention.

Referring to FIG. 12 , in another embodiment of the present invention, the terminal 100 may provide an interactive underground facility-related mixed reality by tracking changes in the position and posture of the terminal 100 . Hereinafter, descriptions overlapping those described above may be summarized or omitted for effective description.

In detail, first, the terminal 100 may control the camera 150 to obtain a captured image and 3D spatial information. (S301)

Here, the terminal 100 may acquire 3D spatial information based on the reference point 10 according to the above-described embodiment.

For example, referring to FIG. 13 , the terminal 100 uses the mark nail installed in the field as the reference point 10, and the mark nail object from the captured image obtained through the camera 150 according to the above-described embodiment. can be detected, and 3D spatial information can be obtained based on the detected nail object.

In another embodiment, the terminal 100 may obtain a 3D spatial coordinate system of 3D spatial information using DGPS (High Precision GPS). That is, the terminal 100 may obtain a three-dimensional spatial coordinate system capable of tracking the location of the terminal in the three-dimensional space of the captured image based on DGPS (High Precision GPS).

Next, the terminal 100 may display a mixed reality (MR) image according to the obtained 3D spatial information. (S303)

In detail, the terminal 100 may obtain scene specification information for the corresponding 3D space based on the obtained 3D spatial information, and may obtain the scene specification information in the 3D space based on the obtained scene specification information and the 3D spatial information. A virtual object matching to can be rendered. In addition, the terminal 100 may display the rendered virtual object as a mixed reality (MR) image by superimposing it on the captured image representing the 3D space.

Also, the terminal 100 may track a change in the position and orientation of the camera 150 according to the movement and posture of the terminal 100 . (S305)

In detail, the terminal 100 first updates the 3D spatial information by tracking changes in the position and orientation of the camera 150 of the terminal 100 based on the acceleration sensor and/or the gyro sensor of the inertial sensor (IMU, 140). can do.

Here, the sensitivity to changes in the position and orientation of the inertial sensor (IMU, 140) is very high, so it is possible to immediately sense information on the change in the position and orientation of the terminal 100 that is minutely changed in real time, and the processor uses the inertial sensor ( Based on the real-time sensing data of the IMU 140 , a change in the position and orientation of the terminal 100 may be tracked and the virtual object may be updated and displayed.

However, the tracking information of the inertial sensor (IMU, 140) may have a lower accuracy due to a spatial tracking cumulative error occurring as data is accumulated, and thus it may be difficult to display the virtual object at an appropriate location.

To prevent this, the terminal 100 may track the position change of the camera 150 based on DGPS (high-precision GPS), and a three-dimensional spatial coordinate system tracked by the inertial sensor (IMU, 140) in real time. By updating, the accumulated error can be corrected.

In addition, in the embodiment, the terminal 100 is a real object obtained through the camera 150 based on the Visual Inertial Odometry (VIO) technology, that is, based on the information recognizing the reference point 10, the camera ( 150), the 3D spatial information is updated based on this, and the accumulated error of the 3D spatial information tracked by the inertial sensor (IMU, 140) can be corrected.

In detail, in this embodiment, the terminal 100 uses a visual inertial odometry (VIO) technology as shown in FIG. 14 in order to more accurately track changes in the posture and position of the camera 150 to update 3D spatial information. Based on the movement of the terminal 100, it is possible to track a change in the position and orientation of the camera 150.

Here, the visual inertial rangefinder technology refers to a technology that analyzes visual information obtained through the camera 150 to determine a location and a direction based on a feature.

That is, the terminal 100 utilizes the recognition information on the reference point 10 (eg, a sign nail and/or a manhole, etc.) which is a real object obtained from the image taken by the camera 150 to the position and orientation of the camera 150 . The change can be determined, and the 3D spatial information can be updated by correcting the tracking information obtained through the inertial sensor (IMU, 140) and the DGPS (high-precision GPS) based on this.

In more detail, first, the terminal 100 may detect the reference point 10 object in the three-dimensional space obtained through the image captured by the camera 150 .

That is, the terminal 100 may perform image deep learning based on the captured image to detect the reference point 10 object that is matched with each underground facility and installed in the field among the real objects identified from the captured image.

In addition, the terminal 100 may update the 3D spatial information based on at least one or more reference points 10 detected in the captured image.

In detail, the terminal 100 may perform tracking of changes in the position and orientation of the camera 150 based on the visual inertial rangefinder technology based on the detected at least one or more reference points 10 .

In the present embodiment, a method of performing position and orientation change tracking of the camera 150 based on the visual inertial rangefinder technology may apply a conventionally known mathematical algorithm, and in the present invention, the camera ( 150) does not limit or limit the algorithm itself for performing position and orientation change tracking.

That is, the terminal 100 may perform position and orientation change tracking of the camera 150 based on the VIO technology based on the sensed at least one or more reference points 10 , and through this, the space-to-camera 150 continuity It is possible to generate tracking correction information based on location and orientation tracking.

Here, the tracking correction information may be information for correcting the tracking information obtained by the inertial sensor (IMU, 140) and/or DGPS (high precision GPS) based on the tracking information obtained based on the VIO technology.

For example, the tracking correction information is the reference point 10 of the captured image according to the position and orientation of the camera 150 acquired by the inertial sensor (IMU, 140) and/or DGPS (high-precision GPS) (eg, a mark nail). ) position, and information including error information between the position of the reference point 10 (eg, a nail) of the captured image according to the position and orientation of the camera 150 obtained by the visual inertial rangefinder technology.

In detail, for example, the terminal 100, the first reference point in the image taken by the camera 150 according to the position and orientation of the camera 150 obtained by the inertial sensor (IMU, 140) and / or DGPS (high-precision GPS) and The position of the second reference point may be obtained based on a three-dimensional matching coordinate system matched to a corresponding three-dimensional space. In addition, the terminal 100 matches the positions of the first reference point and the second reference point in the image taken by the camera 150 according to the position and orientation of the camera 150 obtained by the visual inertial rangefinder technology to the corresponding three-dimensional space. It can be obtained based on the three-dimensional matching coordinate system.

And the terminal 100, the position of the first and second reference points 10 by the inertial sensor (IMU, 140) and / or DGPS (high-precision GPS) and the first and second reference points 10 by the visual inertial rangefinder technology. By comparing the positions, it is possible to determine whether they match exactly.

At this time, the terminal 100, the position of the first and second reference points 10 by the inertial sensor (IMU, 140) and / or DGPS (high-precision GPS) and the first and second reference points 10 by the visual inertial rangefinder technology It is possible to obtain error information indicating the degree of error between the positions by determining the degree of matching between the positions of the .

Subsequently, the terminal 100 that has obtained the error information may generate tracking correction information based on the obtained error information, and based on the generated tracking correction information, an inertial sensor (IMU, 140) and/or DGPS ( It is possible to correct the tracking information for the position and orientation change of the camera 150 generated by the high-precision GPS).

And the terminal 100, which has corrected the tracking information based on the visual inertial rangefinder technology, may update the 3D spatial information based on the corrected tracking information.

In the above, the position of the reference point 10 of the captured image according to the position and orientation of the camera 150 obtained by the inertial sensor (IMU, 140) and/or DGPS (high precision GPS), and the visual inertial rangefinder technology obtained by It is limited to an example of generating tracking correction information based on an error between the positions of the reference point 10 of the captured image according to the position and orientation of the camera 150 and updating the 3D spatial information by correcting the existing tracking information based on this. However, this is only an example of the present invention and is not limited thereto, and tracking the position and orientation of the camera 150 based on the inertial sensor (IMU, 140) and/or DGPS (high-precision GPS) and VIO technology 3 Various embodiments of updating dimensional spatial information may be possible.

Next, the terminal 100, which tracks the position and orientation change of the camera 150 according to the movement of the terminal as described above, and updates the 3D spatial information based on this, is based on the 3D spatial information updated according to the tracking. to update the scene specification information. (S307)

In detail, the terminal 100 may perform an update with scene specification information for a 3D space that matches the updated 3D spatial information based on the updated 3D spatial information.

That is, the terminal 100 may update the scene specification information generated based on the initial 3D spatial information based on the updated 3D spatial information.

In detail, the terminal 100 performs an update of at least one of virtual object position (coordinate) setting information, virtual object orientation setting information, and virtual object scale setting information of scene specification information based on the updated 3D spatial information. can do.

Next, the terminal 100 that has updated the scene specification information may display the rendered mixed reality (MR) image according to the updated scene specification information. (S309)

In detail, the terminal 100 matches and renders a virtual object matching on the three-dimensional space of the captured image based on the updated scene specification information and the initial camera posture and initial camera position of the updated three-dimensional spatial information. )can do.

In more detail, the terminal 100 may determine the camera 150 viewpoint of the terminal 100 based on the initial camera posture and the initial camera position of the updated 3D spatial information, that is, the posture (azimuth) and position of the terminal. In addition, rendering may be performed by matching virtual objects matching the three-dimensional space of the captured image according to the viewpoint of the camera 150 determined based on the updated scene specification information.

In addition, the terminal 100 may output the rendered virtual object by superimposing it on the captured image through the display unit 120 .

As described above, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention integrates data related to underground facilities and mixed reality (MR, Mixed Reality) technology in conjunction with a user's terminal to facilitate management and management of underground facilities By supervising, based on mixed reality technology, it is possible to provide users with 3D visualization of real-time information on underground facilities such as water and sewage, electricity, communication, gas pipes, electrical wiring of buildings, and heating and cooling pipes, and to provide users with underground facilities and buildings. It has the effect of building a mixed reality service infrastructure that provides empirical content specialized for landfill inspection and maintenance.

In addition, the method and system for providing a mixed reality related to an underground facility according to an embodiment of the present invention interact with the user's terminal in real time to update the underground facility information in real time according to a change in the location and/or orientation (attitude) of the user terminal by using mixed reality, there is an effect that it is possible to accurately and conveniently provide information on underground facilities that matches the space in which the user wants to check information on underground facilities.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention utilizes DGPS (Differential Global Positioning System, high-precision GPS) to match and match a three-dimensional space with an underground facility, thereby having improved accuracy. It has the effect of providing information on underground facilities.

In addition, the method and system for providing underground facility-related mixed reality according to an embodiment of the present invention implements image recognition/tracking/learning technology and 3D spatial recognition/camera tracking technology for providing information on underground facilities, thereby providing high quality underground It is possible to provide facility information, which has the effect of maximizing the efficiency of underground facility investigation, analysis, design, prevention of education, and supervision.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention utilizes a camera and/or an inertial sensor (IMU, Inertial Measurement Unit) of the terminal to determine the position and/or orientation (posture) of the user terminal. By providing information on underground facilities according to changes, there is an effect that it is possible to provide information on underground facilities according to changes in the state of the user terminal through its own algorithm and process without a separate location tracking device.

In addition, the method and system for providing mixed reality related to underground facilities according to an embodiment of the present invention is based on the 3D spatial information provided by the national spatial information platform (V-WORLD), the underground facilities buried underground on the electronic map drawing. By mapping, it is possible to provide a more realistic and accurate shape of underground facilities including numerical topography, and to ensure compatibility and performance of existing infrastructure and new software (S/W).

In addition, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be converted into one or more software modules to perform processing according to the present invention, and vice versa.

The specific implementations described in the present invention are only examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings exemplarily represent functional connections and/or physical or circuit connections, and in an actual device, various functional connections, physical connections that are replaceable or additional may be referred to as connections, or circuit connections. In addition, unless there is a specific reference such as “essential” or “importantly”, it may not be a necessary component for the application of the present invention.

In addition, although the detailed description of the present invention has been described with reference to a preferred embodiment of the present invention, those skilled in the art or those having ordinary knowledge in the art will appreciate the spirit of the present invention described in the claims to be described later. And it will be understood that various modifications and variations of the present invention can be made without departing from the technical scope. Accordingly, the technical scope of the present invention should not be limited to the contents described in the detailed description of the specification, but should be defined by the claims.

Claims (7)

A method of providing mixed reality related to underground facilities that manages and supervises facilities buried underground based on data and mixed reality technology related to underground facilities in the processor of the terminal,
controlling the camera to obtain a captured image;
detecting a reference point object by image processing the captured image;
recognizing a code displayed on a reference point in the reference point object;
obtaining 3D spatial model information matching the recognized code;
setting 3D spatial information for the captured image based on the obtained 3D spatial model information; and
and displaying a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the three-dimensional spatial information,
The step of detecting a reference point object by image processing the photographed image includes detecting a circular sign nail that matches an underground facility and is marked with at least one code as the reference point object,
The step of recognizing the code displayed on the reference point in the reference point object includes the steps of detecting the sign nail as an ellipse in the captured image, homography-converting the image of the ellipse region into a circular image, and obtaining homography conversion information and detecting and recognizing a code displayed as embossed or engraved in the converted circular image,
The step of obtaining the three-dimensional space model information matching the recognized code includes detecting information about the location and orientation of the underground facility matched to the sign nail pre-stored with respect to the recognized code,
The step of setting the three-dimensional spatial information for the captured image includes determining an initial camera posture according to the homography transformation information,
After determining the initial camera posture, tracking changes in the position and orientation of the camera according to changes in homography transformation information to update the 3D spatial information; The method further comprising: updating, rendering and displaying the updated virtual object on the captured image.
A method of providing mixed reality related to underground facilities. The method of claim 1,
The obtaining of the three-dimensional space model information matching the recognized code includes obtaining three-dimensional coordinates of the reference point object specifying the three-dimensional space of the captured image.
A method of providing mixed reality related to underground facilities. 3. The method of claim 2,
The step of setting the three-dimensional spatial information for the captured image includes setting a three-dimensional spatial coordinate system for the captured image based on the obtained three-dimensional coordinates of the reference point object
A method of providing mixed reality related to underground facilities. 3. The method of claim 2,
The step of obtaining the three-dimensional spatial model information matching the recognized code further includes obtaining a scale of the reference point object,
The step of setting the 3D spatial information for the captured image includes obtaining reference point scale information based on a size of the reference point object and a pixel displaying the reference point object
A method of providing mixed reality related to underground facilities. 5. The method of claim 4,
The displaying of a mixed reality (MR) image generated by rendering a virtual object on the captured image according to the three-dimensional spatial information may include adjusting the size of the virtual object matching the three-dimensional space of the captured image to the reference point scale. Comprising the step of augmenting the captured image according to the information
A method of providing mixed reality related to underground facilities. delete The method of claim 1,
The step of updating the three-dimensional spatial information by tracking the change in the position and orientation of the camera according to the change of the homography conversion information after determining the initial camera posture,
Updating the 3D spatial information based on real-time sensing data of an inertial sensor (IMU), and using the updated 3D spatial information of a reference point object recognized based on a Visual Inertial Odometry (VIO) technology Comprising the step of correcting the updated 3D spatial information based on the location and direction
A method of providing mixed reality related to underground facilities.

Images