KR20220108911A - Augmented Reality GPS Linked Artificial Intelligence Image Analysis Building Safety System - Google Patents

Abstract

The present invention relates to an augmented reality GPS linked artificial intelligence image analysis building safety system which utilizes building data as open data to allow 911 paramedics to immediately identify the situations (entrance, emergency route, stairway, and window position) of a building in the event a fire with a smartphone in augmented reality, and to quickly extinguish the fire or make residents evacuate the building efficiently, so that it is possible to improve the quality of life and the user side of a smart city.

Description

Augmented Reality GPS Linked Artificial Intelligence Image Analysis Building Safety System

The present invention relates to an augmented reality GPS-linked artificial intelligence image analysis building safety system. By using building data as open data, 119 members of the building are immediately smart It is possible to improve the quality of life and users of the smart city by identifying it with augmented reality with a phone and quickly extinguishing a fire or evacuating residents efficiently.

The fundamental reason why smart cities are attracting attention around the world, including Korea, is their ability to present a blueprint for solving urban problems and to solve problems. Although cities around the world are already suffering from environmental, transportation, energy, and safety due to urbanization, urbanization is already becoming an unstoppable flow. According to a UN report, by 2050, 70% of the world's population will live in cities. In order to solve various problems caused by such urbanization, in the past, efforts such as increasing roads and expanding police personnel were made in the aspects of civil engineering and architecture, but in the technical aspect, ICT was used to inform less clogged roads and CCTVs were used to socialize the society. It is the effectiveness of smart cities to promote safety.

However, although the spread of the underlying technology and hardware and the software control function are growing, the content that provides immediate help to safety accidents to improve the quality of life is still insufficient.

In particular, it is difficult to find products that incorporate ICT for building safety, and it is difficult to find a commercialization model that has built an ecosystem by sharing, adding, and spreading data based on big data. One of the differentiations between U-City and Smart City is the use of open data (public and private).

The present invention aims to solve the above problems, and specifically to provide an augmented reality GPS-linked artificial intelligence image analysis building safety system that can provide AR contents that can promote building safety in the city do.

In addition, the present invention is based on the data of the building, in the event of a fire or a crime, 119 members and police officers immediately check the shape and information of the building through their smartphone through augmented reality, and instantly check the main entrance, stairs, evacuation route, etc. It aims to provide an augmented reality GPS-linked artificial intelligence image analysis building safety system that can provide augmented reality content for easy identification.

In addition, the present invention uses artificial intelligence image reading technology in addition to GPS technology to display the building information (according to open information consent) built with big data in real time in three dimensions, even if the shape or location of the building is not well understood. It aims to provide an augmented reality GPS-linked artificial intelligence image analysis building safety system that can be recognized directly through a smartphone.

In addition, the present invention establishes an ecosystem that can continuously develop through 5G, AR, WebGL, and artificial intelligence so that it can perform the function of 'safety', one of the three major factors and information stability of the smart city, and The purpose is to provide a platform.

According to an aspect of the present invention in order to solve the above problems, a building safety system is provided based on the user's location information and building already information analysis, and the building safety system includes an augmented reality server for providing augmented reality content; a mobile terminal for displaying augmented reality content provided from an augmented reality server; and a communication network connecting the augmented reality server and the mobile terminal, wherein the mobile terminal includes: a camera for taking an image; GPS module for receiving location information; and an augmented reality module for displaying augmented reality content on the captured image of the camera, wherein the augmented reality server includes: an augmented reality content database in which 3D modeling data for a building is stored; and an information providing module, wherein the information providing module of the augmented reality server augments 3D modeling data for a building using any one or both of the user location information data transmitted from the user's mobile terminal or the image data of the camera. It is configured to provide to the user's mobile terminal as real content.

In the above aspect, the information providing module includes: a location-based information providing module for specifying a building photographed by the user based on the user's location information; and an object-based recognition information providing module for specifying the building photographed by the user based on the image of the building photographed by the user.

In addition, in the above aspect, the augmented reality content 3D modeling data may be provided to the user on the web of the mobile terminal or provided through the application of the mobile terminal.

In addition, in the above aspect, the object-based recognition information providing module extracts the detail/feature point from the building image captured from the user's camera, and compares the extracted detail/feature point with the image stored in the database to determine the building that has been transmitted from the user's terminal. Includes building learning data learned using machine learning or neural network artificial intelligence technology to specify.

In addition, in the above-described aspect, the 3D modeling data includes structural information of one or more of an entrance, a staircase, a passage, and an emergency exit of a building.

In addition, in the above aspect, the mobile terminal includes a touch interface, and 3D modeling data displayed on the mobile terminal is configured to be able to zoom in, zoom out, rotate, and move through the touch interface.

The present invention relates to an augmented reality GPS-linked artificial intelligence image analysis building safety system. By using building data as open data, 119 members of the building are immediately smart It is possible to improve the quality of life and users of the smart city by identifying it with augmented reality with a phone and quickly extinguishing a fire or evacuating residents efficiently.

1 is a view showing an example of a building safety system according to the present invention;
Figure 2 is a diagram schematically showing the configuration of the user's mobile terminal of the building safety system according to the present invention;
Figure 3 is a diagram schematically showing the configuration of the augmented reality server in the building safety system according to the present invention;
Figure 4 is a view showing an example of dividing the map data including GSP information into a plurality of areas in the building safety system according to the present invention;
FIG. 5 is a diagram illustrating one area among a plurality of areas of FIG. 4 in more detail.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed herein are only exemplified for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiment according to the concept of the present invention These may be embodied in various forms and are not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention may have various changes and may have various forms, the embodiments will be illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from other components, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named as a second component, Similarly, the second component may also be referred to as the first component.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle. Expressions describing the relationship between elements, for example, “between” and “between” or “directly adjacent to”, etc. should be interpreted similarly.

The terminology used herein is used only to describe specific embodiments, and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that an embodied feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. It can be understood that the same reference numerals in each figure indicate the same member.

1 is a diagram illustrating an example of a building safety system (hereinafter referred to as a building safety system) based on augmented reality GPS and artificial intelligence image analysis according to the present invention.

As shown in Figure 1, the building safety system according to the present invention is an augmented reality server 10 for providing augmented reality contents; and a mobile terminal 40 for displaying augmented reality content provided from the server 10 , wherein the augmented reality server 10 and the mobile terminal 40 are connected to each other through the communication network 20 .

The communication network 20 serves to provide a path for transmitting and receiving information between the mobile terminal 40 and the augmented reality server 10 . That is, in order to provide the augmented reality content to the display unit of the mobile terminal, it corresponds to a data network that enables data transmission and reception between the mobile terminal 40 and the augmented reality server 10 .

For example, the communication network 20 that performs such a function is an IP network that provides a large-capacity data transmission/reception service and a data service without interruption through an Internet Protocol (IP). It may be an All IP network, which is an IP network structure that integrates networks. In addition, the communication network 20 may be one of a wired communication network, a mobile communication network, a Wireless Broadband (Wibro) network, a High Speed Downlink Packet Access (HSDPA) network, a satellite communication network, and a Wi-Fi (Wireless Fidelity) network.

In the present invention, the augmented reality content displayed on the mobile terminal 40 is composed of a web 3D-based (WebGL: Web Graphics Library) and preferably does not require a separate application for displaying the augmented reality content, but through a dedicated application. Augmented reality content for the same building safety may be provided.

For reference, Web3D is used in various fields such as virtual model houses, corporate websites, art galleries, and information systems to implement 3D graphics in real time through a web browser on the Internet.

Methods for implementing 3D on the web include Virtual Reality Modeling Language (VRML), 3D Markup Language for Web (3DMLW), Extensible 3D (X3D), and Web Graphics Library (WebGL). Among them, VRML, 3DMLW, and X3D require an external plug-in, but WebGL does not require a separate plug-in, and its utility is increasing in that rendering can be performed on a web browser.

To implement Web3D using WebGL, it is necessary to convert the data standards and format of 3D content created in various ways into a format that can be displayed in WebGL. Specifically, the point cloud data acquired through scanning forms a mesh, configures UV coordinates, and generates a texture image that matches the UV coordinates. In addition, 3D modeling data generated through general 3D modeling tools such as SketchUp and BIM (Building Information Modeling) tools such as Revit are converted to the data format to obtain *. and link it to WebGL.

Currently, WebGL runs on Google Chrome, Internet Explorer 11, and development releases of Mozilla Firefox 4, Safari and Opera. Also, as WebGL is supported in the PR1.2 firmware update of Nokia N900, it has the advantage of being widely used.

When a user takes a picture of a building using a camera function provided in a web browser installed in the mobile terminal 40, for example, Google Chrome, Internet Explorer 11, etc. and transmits it to the augmented reality server 10, the server 10 is sent to the corresponding building. The augmented reality content for the mobile terminal 40 is returned to the mobile terminal 40, and the augmented reality content is overlapped and displayed on the building of the image taken from the user's camera.

2 is a diagram schematically illustrating the configuration of the mobile terminal 40 . The mobile terminal 40 provides the user with a function for outputting a 3D augmented reality image on a web browser. As shown in FIG. 2 , the mobile terminal 40 includes a control unit 41 , a camera 42 , an interface 43 , an input unit 44 , a storage unit 45 , a GPS module 46 , and a motion controller 47 . and a communication unit 48 and the like. However, the mobile terminal 40 shown in FIG. 2 is not composed of only essential components, and the mobile terminal 40 of this embodiment may further include other components, and some of the illustrated components may be removed. and may be provided as an integrated component.

The mobile terminal 40 is a device that receives augmented reality content from the server 10 and displays the received augmented reality content on the display unit of the terminal 40 so that the user can use the augmented reality content. It includes a GPS module 46 for identifying a current location, a camera 42 for photographing a building, and a display unit capable of outputting augmented reality content.

The storage unit 45 includes a program for processing and control of the control unit 41 (eg, an algorithm for matching an image frame and a 3D augmented reality image); Augmented reality data that matches the building; includes, and also serves for temporary storage of input/output data.

The storage unit 45 is a flash storage type, a hard disk type, a multimedia card micro type, a card type storage unit (SD, etc.), a RAM (RAM), a static RAM (SRAM), a ROM (ROM), an EEPROM (Electrically Erasable) The storage medium may be any one of a programmable ROM), a programmable ROM (PROM), a magnetic storage unit, a magnetic disk, and an optical disk.

The camera 42 is used as a tool for photographing a building. In addition, the mobile terminal 40 may further include a display unit including a touch screen, a speaker, an alarm unit, and a haptic module as components for generating input/output related to visual, auditory or tactile sense. When the touch screen is included, the user can zoom in, zoom out, rotate, and move the augmented reality data (3D modeling data) provided on the touch screen.

The display unit may display information governed by the control unit, display an image frame photographed by the camera 42, or display a screen in which 3D augmented reality data is matched to the photographed image frame.

The communication unit 48 may include any one or more such as a broadcast reception module, a mobile communication module, a wireless Internet module, a short-range communication module, and a location information module, and is augmented through the communication network 20 using the communication unit 48 . Various data for realization may be requested or transmitted from the server 10 . Here, the wireless Internet module refers to a module for wireless Internet access, and may be built-in to the mobile terminal 100 or configured to be externally connectable. As wireless Internet technologies, wireless LAN (WLAN) (Wi-Fi), wireless broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), etc. may be used.

The control unit 41 is configured to control the overall operation of the mobile terminal 40 . For example, the control unit may transmit the user's GPS information or the image captured by the user to the server 10 through the communication unit 46 to request corresponding augmented reality data or 3D modeling data.

In addition, the control unit 41 matches the image frame including the image captured by the camera 42 with the 3D augmented reality data, and outputs it to the display unit of the mobile terminal 40 .

In addition, the controller 41 may operate to enlarge, reduce, rotate, or move the 3D augmented reality data displayed on the user's mobile terminal in response to a user input (input on the touch screen).

Also, a motion controller 47 may be connected to the controller 41 . The motion controller 47 may be a gyro sensor, an acceleration sensor, or a GPS sensor installed in the mobile terminal. That is, when a compass module, a gyro sensor, an acceleration sensor, a GPS sensor, etc. are installed in the mobile terminal 40, the controller 41 determines the direction of the mobile terminal 40 using the compass module, or 40), it is also possible to control the display of the 3D augmented reality data in response to the shaking or tilting motion.

Also, the mobile terminal 40 further includes a GPS module 46 . The GPS module 46 may calculate its current location, and the controller 41 may provide the calculated GPS value to the augmented reality server 10 to request augmented reality data corresponding to the GPS value.

3 is a diagram schematically illustrating an internal configuration of the augmented reality server 10 shown in FIG. 1 . As shown in FIG. 3 , the mobile terminal 40 and the communication unit 11 for transmitting and receiving data for providing augmented reality data (3D modeling data), and augmented reality content in which the augmented reality data (3D modeling data) is stored DB(14); Based on the GPS information received from the mobile terminal 40 or the image received from the mobile terminal, the augmented reality data matched thereto is found from the database 14 and the information providing module 13a for notifying it to the controller !2 , 13b).

When the control unit 12 receives a request for augmented reality content for a specific building from the mobile terminal 40 through the communication unit 113, the controller 12 uses any one or both of the information providing modules 13a and 13b to discover the augmented reality content. and transmit the found content to the mobile terminal 40 .

The location-based information providing module 13a includes location-based information and related services, and in one embodiment, the location-based information providing module includes Google's Android mobile API components as follows: (1) Access_Fine_Location, (2) This can be done by integrating Location Provider, (3) Location Listener, (4) Location Updates, and (5) Reverse GeoCodings.

3 and 4 are diagrams for explaining the operation of the location-based information providing module 13a. 3 and 4 , the location-based information providing module 13a includes map information GI (or map data), and one map data is divided into a plurality of regions. Since each area (the first to fifth areas) includes GPS information, as shown in FIG. 3 , when the user transmits his or her location information (latitude: 5400, longitude: 1041) to the server, the location of the server The base information providing module 13a discovers information on an area in which the user is located from the user's location information, and transmits the information on the area in which the user is located to the controller 12 .

After the user's area is determined based on the location-based information, the object-based recognition information providing module includes discovering a designated object (building) in an image or video sequence in computer vision. In one embodiment, object-based recognition performs the following actions: (1) extracting details/features from a live view stream received from the user's terminal, (2) comparing with a photo database stored in the database, and (3) delivering the result immediately. This can be done by using an algorithm that implements

That is, the object-based recognition information providing module extracts detail/feature points from the building image captured from the user's camera, compares the extracted detail/feature points with the image database stored in the database, and has been transmitted from the user's terminal. building is specified. To this end, the object-based recognition information providing module may include building learning data learned using machine learning or neural network artificial intelligence technology.

Machine learning or neural networks perform computations for inference and prediction according to artificial intelligence (AI) technology. Specifically, the neural network may be an artificial neural network (ANN) or a deep neural network (DNN) that performs an operation through a plurality of layers. A neural (neural network) network is a deep neural network (DNN) when the number of layers is plural according to the number of internal layers performing an operation, that is, when the depth of a neural network performing an operation increases. ) can be classified as In addition, the deep neural network (DNN) operation may include a convolutional neural network (CNN) operation and the like. That is, the object-based recognition information providing module may implement a building estimation model through the exemplified neural network, and learn the implemented building estimation model using learning data. Then, the image of the building inputted through the learned building estimation model is analyzed and estimated.

The building learning model used in the present invention is PLSR (Partial Least Square Regression), PLS (Partial Least Square), VIP-PLSR (Variable Importance in the Projection-Partial Least Square Regression), Full-ANN (Full-Artificial Neural Networks) ), an artificial neural network (ANN), a partial least-squares and artificial neural network (PLS-ANN), and a convolution neural network (CNN) may be used, but the present invention is not limited thereto.

After the estimation of the building is made in the object-based recognition information providing module 13b, the object-based recognition information providing module notifies the control unit 12 of the unique identifier of the building. And the control unit 12 finds the 3D modeling data of the corresponding building from the augmented reality content DB 14 in which the 3D modeling data of the building is stored using the unique identifier of the building, and uses it to find the 3D modeling data of the building in the augmented reality module 41a of the mobile terminal 40 will be sent to

Afterwards, the augmented reality module 41a of the mobile terminal 40 matches and displays the 3D modeling data, which is the augmented reality data transmitted from the augmented reality server 10, on the image of the building displayed on the user's mobile terminal 40. .

Although the above-described embodiment has been described as an example using a combination of the location-based information providing module 13a and the object-based recognition information providing module 13b, the present invention is not limited thereto. can be used to provide 3D modeling data to users. For example, when the location-based information providing module 13a is used alone, 3D modeling data for a building located closest to the user's location may be provided using the user's GPS information, and object-based recognition information is provided. When the module 13b is used alone, 3D modeling data for a building most similar to a feature point included in a building image received from a user's camera may be provided.

The 3D modeling data is displayed overlapping the building displayed on the user's screen, including structural information of the building such as entrances, stairs, passages, and emergency exits of the building. It is provided so that the structure can be recognized immediately through the terminal, so that it is possible to provide a safer environment when firefighters or police officers enter the building.

In addition, since the 3D modeling data displayed on the user's mobile terminal is configured to be enlarged, reduced, rotated, and moved through manipulation of the touch screen, the internal structure of the building can be checked more precisely.

In addition, the 3D modeling data of the building is configured to track the exact location of the building from any angle by recognizing a 360-degree image pattern, and to reduce/enlarge the building information in 1:1 ratio and 1:n ratio. .

In addition, by using the gyro sensor of the user terminal as an input to display 3D modeling data, it is possible to display the main location in the same way as the building seen in the real field of view.

The device described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that may include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the limited drawings as described above, various modifications and variations are possible by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

10: augmented reality server 20: communication network
40: mobile terminal 41: control unit
42: camera 43: interface
44: input unit 45: storage unit
46: GPS module 47: motion controller
48: communication unit (mobile terminal) 11: communication unit (server)
12: control unit 13: information providing unit
14: Augmented reality content DB

Claims (6)

In the building safety system based on user's location information and building already information analysis,
Augmented reality server for providing augmented reality content;
a mobile terminal for displaying augmented reality content provided from an augmented reality server; and
Including a communication network connecting between the augmented reality server and the mobile terminal,
The mobile terminal may include: a camera for taking an image; GPS module for receiving location information; and an augmented reality module for displaying augmented reality content on the captured image of the camera,
The augmented reality server may include: an augmented reality content database in which 3D modeling data for a building is stored; and an information providing module;
The information providing module of the augmented reality server uses any one or both of the user location information data transmitted from the user's mobile terminal or the image data of the camera to provide 3D modeling data for the building as augmented reality content to the user's mobile terminal. characterized in that it is configured to provide
building safety system.
According to claim 1,
Information provision module,
a location-based information providing module for specifying a building photographed by a user based on the user's location information; and
Characterized in that it comprises an object-based recognition information providing module for specifying the building photographed by the user based on the image of the building photographed by the user.
building safety system.
According to claim 1,
The 3D modeling data, which is the augmented reality content, is provided to a user on a web of a mobile terminal or a building safety system provided through an application of a mobile terminal.
3. The method of claim 2,
The object-based recognition information providing module extracts detail/feature points from the building image captured from the user's camera, and compares the extracted detail/feature points with the image stored in the database to specify the building that has been transmitted from the user's terminal. Building safety system that includes building learning data learned using neural network artificial intelligence technology.
According to claim 1,
3D modeling data includes structural information of one or more of the building's entrances, stairs, passages, and emergency exits.
building safety system.
6. The method of claim 5,
The mobile terminal includes a touch interface, and the 3D modeling data displayed on the mobile terminal is configured to be able to zoom in, zoom out, rotate, and move through the touch interface.
building safety system.

Images