TWM560099U - Indoor precise navigation system using augmented reality technology - Google Patents

Abstract

An indoor precision navigation system using augmented reality technology includes: a mobile device, a camera device, a screen, a wireless communication module, a GPS module, an inertial measurement unit, and an augmented reality a module and an arithmetic processor; a cloud server, and a wireless communication module for transmitting data and storing a building map, the figure includes a plurality of positioning points, some of the positioning points are set as selected landmarks; the camera device takes a film and inertial measurement The value of the unit is estimated by the arithmetic processor in the spatial position possible in space. The measurement error accumulated by the continuous motion is corrected by the selected landmark, autoregressive filter and nonlinear optimizer, and the spatial orientation is mapped to the building map. The camera device obtains the augmented virtual information from the augmented reality module and displays it on the screen for navigation.

Description

Indoor precision navigation system using augmented reality technology

The present invention relates to a navigation system, and more particularly to an indoor precision navigation system using augmented reality technology.

In the technical field of Augmented Reality (AR) applied to indoor navigation, one of the existing main technologies obtains preliminary positioning information through the GPS positioning module, and then uses the attitude information obtained by the inertial sensing module as positioning. For reference, the preset virtual reference information is captured on the assumed positioning reference point, and the virtual information with the navigation message is mapped onto the screen displaying the actual scene, but after the indoor GPS signal is lost, the inertial sensing module is obtained. The result of the attitude information calculation will produce an accumulated error, so it is necessary to reuse the pre-positioning method such as Marker or Natural Features to assist the calculation in order to obtain a more accurate hypothetical reference point coordinate. The virtual information corresponds correctly. Appearing on the screen, this method requires the provision of a plurality of markers such as positioning facilities in the building, thereby increasing the overall construction cost of the navigation system.

Another technique is to perform optical flow analysis by using a plane image captured by an image unit as a method of positioning measurement, for example, the invention patent of the navigation system of the augmented reality technology No. I574223 of the Republic of China. The invention uses the continuous random image capture to perform optical flow analysis and the reference pose module calculates a displacement vector, and adds a displacement vector from the start coordinate obtained from the QR Code or the RFID, and corresponds to the map data. The coordinate system obtains an estimated coordinate to output a positioning signal related to the estimated coordinate. Since the calculation of the optical flow analysis method still has an error value, the method does not need to specifically set a plurality of positioning facilities in the building to save the overall construction cost of the navigation system, but the farther away from the starting coordinate, the more accumulated. The large error value makes the estimated coordinates different from the actual position, and it is impossible to accurately navigate to the correct position and thus cannot obtain the virtual information provided by the augmented reality.

One of the aims of this creation is to provide an indoor precision navigation system that can reduce the setting of positioning facilities and utilize augmented reality technology to improve indoor positioning accuracy.

In order to achieve the above objective, the present invention uses an augmented reality technology indoor precision navigation system, which in one embodiment comprises: a mobile device, a built-in camera, a screen, a wireless communication module, a GPS module, and a An inertial measurement unit, an augmented reality module and an arithmetic processor; and a cloud server capable of transmitting data with the wireless communication module, the cloud server storing a building map, the building map The method includes a plurality of positioning points, the plurality of positioning points including actual GPS coordinates and floor information, wherein: the plurality of positioning points are partially set as selected landmarks, and the selected landmarks are digital images pre-selected and stored in the building or Digital information; the mobile device filters out the image of the vicinity of the GPS point according to the last coordinate of the GPS and downloads the image of the building; the film is continuously used to capture the film, and the film in the film passes through the computing processor to visualize the computer Point extraction and tracking to estimate the possible continuous motion pose and continuous motion vector of the mobile device in space, while measuring the number measured by the inertial measurement unit The motion analyzer and the inertia integrator are used to calculate the possible continuous motion posture and the continuous motion vector of the mobile device, and the result of the projection between the imaging device and the inertial measurement unit is integrated, and the truncated length complement is used to calculate the current possibility. Spatial positioning, and through the selection of landmarks, autoregressive filters and nonlinear optimizers to correct the measurement error accumulated in the three-dimensional continuous motion, the spatial positioning of the mobile device corresponds to the downloaded building map, and finally Augmented virtual information is acquired from the augmented reality module in the real 3D space of the camera and displayed on the screen for navigation.

In some embodiments of the present invention, when the GPS module of the mobile device receives the GPS signal indoors, the GPS location information is transmitted to the cloud server, and the cloud server searches for the indoor location near the GPS coordinate. The map ignores the coordinates of other coordinates to improve the accuracy of the search and filter the search range to improve the response time of the search.

In some embodiments of the present invention, wherein the cloud server stores the building map, the building map of a building is nested on a virtual square of the editor, and the building map is scaled and adjusted to an appropriate scale. The true size of the building is known by the proportion of the square.

In some embodiments of the present invention, wherein the inertial measurement unit detects that the mobile device is walking or going up the stairs or taking the elevator or the user lowering the mobile device or following the hand swing or the head posture or the body posture. Feedback and correction are performed to eliminate the image captured by the camera to obtain accurate positioning and navigation estimation.

In some embodiments of the present author, wherein the selected landmarks comprise pre-photographed building images or particular drawings or pre-set bar codes.

In some embodiments of the present invention, the positioning device performs the positioning at the starting point by searching for a known image captured in advance near the positioning point as a landmark, and comparing the features captured from the landmark or the positioning point image. Use the augmented reality related computer algorithm to know your current indoor positioning, and then download the map of the positioning point.

The present invention is not limited by the embodiments, but is not intended to limit the scope of the present invention. In order to make the purpose, features and advantages of the present invention clear, the preferred embodiment will be described below. In conjunction with the detailed description of the drawings, certain features of the present specification, which are described in some separate embodiments, may also be implemented separately or in combination in other embodiments. Conversely, various described in the context of the various embodiments. Features may also be implemented in other separate or multiple embodiments, either separately or in any suitable sub-combination.

Referring to FIG. 1, a block diagram of an indoor precision navigation system using augmented reality technology is created. An indoor precision navigation system using augmented reality technology includes a mobile device 10 in a preferred embodiment. The mobile device 10 includes a camera device 11, a screen 12, a wireless communication module 13, a GPS module 14, an inertia measurement unit 15, an arithmetic processor 16, and an augmented reality module 17; a cloud server 20, the cloud server 20 can transmit data with the wireless communication module 13, the cloud server 20 stores a building map 21, and the building map 21 includes a plurality of anchor point (POI) information. The plurality of anchor point information includes actual GPS coordinates and floor information.

The mobile device 10 is preferably a mobile phone, a tablet computer, a notebook computer, a headset, a Google glasses, or a mobile device. The mobile device 10 includes at least one camera device 11 and a screen. 12. A wireless communication module 13, a GPS module 14, an inertia measurement unit 15, an arithmetic processor 16, and an augmented reality module 17 for performing augmented reality technology and navigation technology implementation, The camera device 11 can capture an image through an optical lens and digitally output the image; the screen 12 can display an image obtained by the camera 11 and a screen or an operation interface to be displayed by the system; the mobile device 10 can pass through the wireless communication module 13 and other devices. The data transmission is performed; the GPS module 14 can obtain the position information from the global satellite positioning system for positioning; the inertial measurement unit 15 (IMU) can calculate the posture of the object, that is, solve the posture of the mobile device 10; The arithmetic processor 16 can perform various operations, and the augmented reality module 17 can display the augmented reality data in combination with the image of the camera 11 on the screen 12 Line navigation.

The cloud server 20 can transmit data to the wireless communication module 13 through a Wifi, Bluetooth, mobile communication, and various wireless communication technologies. The cloud server 20 stores a building map 21, and the building map 21 It includes several Point of Interest (POI) information, which includes actual GPS coordinates and floor information.

The plurality of positioning points are selected points of the navigation system, and the positioning points may be any points of interest in the building that may be sought, such as: departments, units, elevator positions, restrooms, service desks, department store counters, Hitchhiking points, platforms, exits, taxis, etc., etc., in which the design specifically sets some of the positioning points as Landmarks, which are preferably points with unique images in the building, such as There is a painting in the building, which can be directly taken or stored in the digital image file of the painting; or, for example, there is a poster in the building, which can be directly photographed to store the image, or the digital number of the poster can be used. Image file; or, for example, a special line or element in a building to store the digital image captured by it; or preset a one- or two-dimensional bar code to store the digital information represented by the bar code; pre-select and store the image The digital image and digital information are set as selected landmarks, and the image of the selected landmark is stored in the cloud server 20.

When the navigation system is turned on, firstly locate a starting point of the mobile device 10 at a location where the building can receive the GPS signal, and according to the last coordinate of the GPS, filter out the map of the vicinity of the GPS point and download the building map (for example, , filtering out the image within 300 meters, and then using the feature comparison to find the most likely positioning); the mobile device 10 uses the camera 11 to continuously shoot the film during the journey, and the frame in the film passes through the operation processor. 16 extracting and tracking the possible moving motion posture and continuous moving vector of the mobile device 10 in the space with the extraction and tracking of the computer visual feature points, and calculating the value measured by the inertial measurement unit 15 through the motion analyzer and the inertia integrator The possible continuous motion posture and the continuously moving vector of the mobile device are integrated with the results of the estimation of both the imaging device and the inertial measurement unit, and the truncated length complement is used to calculate the most probable spatial positioning; when the imaging device 11 captures the special selection When the landmark is used, the selected landmark is identified and the three-dimensional continuous motion is corrected by the selected landmark, autoregressive filter and nonlinear optimizer. The accumulated measurement error makes the positioning more precise, and the spatial positioning of the mobile device 10 corresponds to the downloaded building map 21, and finally, in the real 3D space of the camera device 11, the augmented reality module 17 Acquire augmented virtual information and display it on the screen 12 for navigation.

Referring to FIG. 2, FIG. 2 is a flowchart of a management system for constructing an indoor positioning map of the indoor precision navigation system using the augmented reality technology; in the embodiment, the cloud server 20 stores the building. Figure 21 is established by the following steps: S201 puts the floor plan into the virtual grid; stacks the building plan of a building on the virtual square of the editor, S202 scales the floor plan and fits the actual size; And adjusting to an appropriate ratio, the real size of the building is known by the scale of the square, and S203 sets a point of interest (POI) in the virtual grid; that is, several positioning points included in the aforementioned building map 21 . S204 uses GPS localtion to put the virtual grid into the global map; uses the GPS position to put the virtual grid that has been placed in the floor plan of the building into the global map, S205 rotates the virtual grid in the correct direction, and S206 sets the virtual grid to the building. Floor, S207 uses the image or reference identifier to set the landmark (LandMark); pre-selected and stored digital images or digits such as specific maps (such as specific drawings or special building images or barcodes) in some buildings. The information is set to a Landmark, and S208 saves the virtual grid.

Referring to FIG. 3, it is a flowchart of a mobile device positioning system of an indoor precision navigation system using augmented reality technology; when performing indoor precise navigation, first step S401 is performed to put the device into an indoor environment; the user will hold the hand or the head. The mobile device 10 is placed in a known indoor environment for positioning, and the latest GPS coordinates are sent to the cloud location server in step S402; since some indoors can still receive GPS signals, if they are received, they can be pre-stored. Taking the position obtained by the GPS signal as a basis, the cloud server 20 can search for indoor positioning maps near the GPS coordinates, ignoring the coordinates of other coordinates to improve the accuracy of the search; in the embodiment, when the GPS of the mobile device 10 The module 14 transmits the GPS location information to the cloud server 20 when the GPS signal is received indoors, and the cloud server 20 searches for the indoor location map near the GPS coordinate to ignore the coordinates of other coordinates, thereby improving the accuracy of the search. Degree, and filter the search range to improve the search response time to improve performance, step S403 sends the instant camera stream image to the cloud The location server; the instant image of the camera device 11 and the last possible GPS coordinates of the mobile device 10 are sent to the cloud server 20 for comparison. The cloud server 20 uses the last possible GPS coordinates transmitted by the mobile device 10 to narrow the range and find nearby possibilities. The selected landmarks are compared to the features selected from the selected landmarks, and the augmented reality related computer algorithms are used to know their current indoor positioning, and then the maps near the anchor points are downloaded.

Please refer to FIG. 4 and FIG. 5 at the same time. FIG. 4 is a block diagram of the indoor positioning system of the indoor precision navigation system using the augmented reality technology. FIG. 5 is the cloud of the indoor precision navigation system using the augmented reality technology. The flow chart of the server positioning system; the flow of the cloud server 20 to perform the positioning system is as follows: S601 analyzes the camera stream image from the device; the instant stream 501 of the device is a film continuously captured by the mobile device 10 through the camera device 11 The instant stream 501 is sent to the stream analyzer 502 for real-time data analysis, and the S602 searches for the nearby indoor map using the latest coordinates of the GPS; the latest coordinates of the GPS 503 and the data analyzed by the stream analyzer 502 are transmitted to the nearest GPS. The neighboring map searcher 504, S603 uses the features of the camera stream image to search for a large number of features stored in the cloud in the result of the previous step; the data analyzed by the stream analyzer 502 also searches for the nearest neighbor to the GPS through a large number of feature point features. The map searcher 504 transmits to the feature analyzer 506 to extract features and transfer them to the feature store 507, simultaneously to the landmarks The identifier 512 is stored for the indoor map storage 513. S604 finds the indoor location of the most similar feature; the path search 514 is stored by the indoor path analyzer 515 to the path store 516, S605 confirms the landmark; the landmark management 508 is stored to the feature store by the feature extractor 509, the graphical indexer 510, and the graphics classifier Body 507. S606 sends the coordinates and the map to the device; the coordinates and the map are transmitted by the cloud server 20 to the mobile device 10.

In some embodiments of the present invention, the selected landmark includes a pre-captured or stored building image or a specific picture or digital information. Referring to FIG. 6, the mobile device of the indoor precision navigation system using the augmented reality technology is created. A schematic diagram of a selected landmark is obtained from a specific picture in the building; when the mobile device 10 uses the camera 1 to continuously capture the specific picture L1, the specific picture L1 is previously calibrated to the GPS coordinates and transmitted to the indoor map storage. Therefore, the indoor position of the most similar feature is found by the specific picture L1, and the coordinates and the map are transmitted to the mobile device 10, and the relevant augmented reality information is displayed on the screen 12 using the augmented reality related computer algorithm.

Referring to FIG. 7, a schematic diagram of a selected landmark is obtained from a special building image in a building using a mobile device of an indoor precision navigation system using augmented reality technology; and a moving image continuously captured by the mobile device 11 by the mobile device 11 The special building image L2 appears, and the special building image L2 has a unique wall line in the building as shown in the figure. Since the special building image L2 is previously calibrated with GPS coordinates and transmitted to the indoor map storage, it is stored. The indoor location of the most similar feature is found by the particular picture L1, and the coordinates and map are sent to the mobile device 10, and the augmented reality related computer algorithm is used to display the relevant augmented reality information on the screen 12.

Referring to FIG. 8 , the technical architecture and data flow description diagram of the navigation algorithm of the indoor precision navigation system using the augmented reality technology is used, and the user takes the camera of the mobile device 10 such as the handheld or the headset after obtaining the initial positioning point. The device 11 continues to capture the film, and the frames in the film pass through the operations of the feature extractor 509, the feature tracker 517, and the camera pose estimator 518, and the values measured by the inertial measurement unit 15 pass through the motion analyzer 523 and the inertial integrator 525. The operation result is integrated through steps such as autoregressive filter 524, nonlinear optimizer 519, cloud position search 520, and map position map 521, and finally enhances the real environment 522 in the real 3D space to augment the virtual information. The specific description is as follows: the mobile device 10 continuously captures a movie by using the camera 11 during the traveling, and the video frame in the movie is used by the computing processor 16 to extract and track the computer visual feature points to calculate that the mobile device 10 may be in space. The continuous motion posture and the continuously moving vector, while the value measured by the inertial measurement unit 15 passes through the motion analyzer 523 and the habit The sex integrator 525 derives a possible continuous motion posture and a vector of continuous movement of the mobile device 10, integrates the results of both the imaging device 11 and the inertial measurement unit 15, and truncates the length to calculate the most probable spatial positioning. Since the operations of the feature extractor 509, the feature tracker 517, the camera pose estimator 518, the motion analyzer 523, and the inertia integrator 525 may generate progressive errors, when the camera 11 captures a selected landmark, the selected landmark is recognized. And through the selected landmark, the autoregressive filter 524 and the nonlinear optimizer 519, the measurement error accumulated in the three-dimensional continuous motion is corrected to make the positioning more precise, and the spatial positioning of the mobile device 10 and the downloaded building map are provided. Corresponding to 21, finally, in the real 3D space of the imaging device 11, the augmented virtual information is acquired from the augmented reality module 17 and displayed on the screen 12 for navigation.

In addition, in this embodiment, the inertial measurement unit 15 can also detect that the mobile device 10 is walking or going up the stairs or sitting on the elevator or the user puts down the mobile device 10 or swings with the hand or the head posture or body posture. The state is analyzed by the motion analyzer 523, and feedback and correction are performed to exclude the camera 11 from capturing the image interfered by the noise to obtain an accurate positioning and navigation estimation.

In summary, the present invention uses the mobile device 10 to perform the positioning at the starting point. The real-time image of the camera device 11 is used to compare the last possible GPS coordinates of the mobile device 10 to the cloud server 20, and the cloud server 20 utilizes the mobile device. 10 The last possible GPS coordinates transmitted, narrow the range to find possible special landmarks nearby, use the search for known images taken in advance near the anchor point as the selected landmarks, and compare the features of the selected landmarks or the anchor images. The augmented reality-related computer algorithm knows its current indoor positioning, and then downloads the image of the anchor point and the augmented reality information displayed on the screen 12. This indoor precision navigation system can reduce the positioning facility setting by using the augmentation The technology of the environment can improve the accuracy of indoor positioning.

The embodiments described above are merely illustrative of the technical ideas and features of the present invention, and the purpose thereof is to enable those skilled in the art to understand the contents of the present invention and implement them according to the scope of the patent. Any changes or modifications made in accordance with the spirit of this creation and the contents of the manual shall be covered by this creation patent.

10‧‧‧Mobile devices
11‧‧‧ camera
12‧‧‧ screen
13‧‧‧Wireless communication module
14‧‧‧GPS module
15‧‧‧Inertial measurement unit
16‧‧‧Operation processor
17‧‧‧Augmented Reality Module
20‧‧‧Cloud Server
21‧‧‧Building map
501‧‧‧ Instant streaming of devices
502‧‧‧Streaming Analyzer
503‧‧‧ GPS's newest coordinates
504‧‧‧GPS nearest neighbor map searcher
505‧‧‧Many feature search
506‧‧‧Feature Analyzer
507‧‧‧Characteristic memory
508‧‧·landmark management
509‧‧‧Feature Extractor
510‧‧‧Graphic indexer
511‧‧‧Graphic classifier
512‧‧‧landmark identifier
513‧‧‧ indoor map storage
514‧‧‧Path Search
515‧‧‧ Indoor Path Analyzer
516‧‧‧Path memory
517‧‧‧Feature Tracker
518‧‧‧ Camera Attitude Estimator
519‧‧‧Nonlinear optimizer
520‧‧‧Cloud location search
521‧‧‧Map location mapping
522‧‧‧Enhance the real environment
523‧‧‧Sports Analyzer
524‧‧‧Autoregressive filter
525‧‧‧Inertial Integrator
L1‧‧‧ specific picture
L2‧‧‧Special Building Image
S201~S208‧‧‧Steps
S401~S403‧‧‧Steps
S601~S606‧‧‧Steps

Figure 1 is a block diagram of an indoor precision navigation system using augmented reality technology. Figure 2 is a flow chart of the management system for constructing an indoor positioning map of the indoor precision navigation system using the augmented reality technology; This is a block diagram of the mobile device positioning system of the indoor precision navigation system using the augmented reality technology; Figure 4 is a block diagram of the indoor positioning system of the indoor precision navigation system using the augmented reality technology; A flow chart of a cloud server positioning system using an augmented reality technology for an indoor precision navigation system; FIG. 6 is a mobile device using an augmented reality technology for an indoor precision navigation system to obtain a selected landmark from a specific picture in a building. Schematic diagram; Figure 7 is a schematic diagram of the selected landmarks of the special building image in the building using the mobile precision indoor navigation system; Figure 8 is the indoor precision of the creation using the augmented reality technology. The technical architecture and data flow diagram of the navigation algorithm of the navigation system.

Claims (6)

An indoor precision navigation system using augmented reality technology, comprising: a mobile device, a built-in camera device, a screen, a wireless communication module, a GPS module, an inertial measurement unit, and an augmented reality a module and an arithmetic processor; and a cloud server capable of transmitting data with the wireless communication module, the cloud server storing a building map, the building map includes a plurality of positioning points, and the plurality of positioning The point includes the actual GPS coordinates and floor information, and is characterized in that: some of the plurality of positioning points are set as selected landmarks, and the selected landmarks are digital images or digital information pre-selected and stored in the building; the mobile device is based on the last GPS a coordinate image of the building in the vicinity of the GPS point; the film is continuously captured by the camera device, and the image in the film is used by the computing processor to extract and track the computer visual feature point to calculate the possible space of the mobile device in the space. a continuous motion posture and a continuously moving vector, and the value measured by the inertial measurement unit is calculated by a motion analyzer and an inertia integrator The possible continuous motion posture and the continuously moving vector of the mobile device integrate the results of the imaging device and the inertial measurement unit, calculate the relative position of the mobile device and calculate the current most probable space. Positioning, and correcting the measurement error accumulated by the continuous motion of the three-dimensional space through the selected landmark, an autoregressive filter and a nonlinear optimizer, and correspondingly spatially locating the mobile device with the downloaded image of the building Finally, the augmented virtual information is obtained from the augmented reality module in the real 3D space of the camera device and displayed on the screen for navigation. If the GPS module of the mobile device receives the GPS signal indoors, the GPS location information is transmitted to the cloud server, and the cloud server searches for the GPS coordinates. The nearby indoor positioning map ignores the coordinates of other coordinates to improve the accuracy of the search and filter the search range to improve the response time of the search. For example, in the request 1, the augmented reality technology of the indoor precision navigation system, the cloud server stores the building map, the building plan of a building is stacked on a virtual square of the editor, and the building is scaled. The map is adjusted to the appropriate scale, and the true size of the building is known by the scale of the square. An indoor precision navigation system using the augmented reality technology of claim 1, wherein the mobile device detects that the mobile device is walking or going up the stairs or taking the elevator or the user lowering the mobile device or swinging or moving with the hand The state of the posture or the posture of the body is fed back and corrected to exclude the camera from capturing the image interfered by the noise to obtain an accurate positioning and navigation estimation. An indoor precision navigation system utilizing augmented reality technology, such as any one of claims 1-4, comprising a pre-captured special building image or a particular picture or bar code. The indoor precision navigation system of the augmented reality technology is used in claim 5, wherein the mobile device performs a positioning system at a starting point by searching for a known image captured in advance near the positioning points as a selected landmark, and the comparison is performed. From the characteristics of the landmark or the image of the anchor point, use the computer algorithm of the augmented reality to know its current indoor positioning, and then download the map of the anchor point.