JP2023509099A - INDOOR VISUAL NAVIGATION METHOD, APPARATUS, SYSTEM AND ELECTRONIC DEVICE - Google Patents

Abstract

In the present disclosure, a mobile device uploads a collected indoor image of a positioning target to a server, the server determines a camera pose when the mobile device collects the indoor image, and the camera pose corresponding to the indoor image is transmitted to the mobile device. After that, the mobile device creates an AR coordinate system that matches the world coordinate system based on the camera pose corresponding to the indoor image, and based on the destination information set by the user, the pre-imported indoor topology map. plan the shortest route in , and finally display the current preview image collected by the mobile device on the screen, and superimpose the 3D mark indicating the route traveling direction on the current preview image based on the AR coordinate system and the shortest route. A matching indoor visual navigation method, apparatus, system and electronic equipment are provided. The present disclosure can provide users with indoor navigation services to guide them to reach their destinations conveniently. [Selection drawing] Fig. 2

Description

The present disclosure relates to the technical field of image processing, and more particularly to indoor visual navigation methods, devices, systems and electronic devices.

<Cross reference to related applications>
This disclosure claims priority to a Chinese patent application entitled “Indoor visual navigation method, device, system and electronic device” filed with the Chinese Patent Office on April 14, 2020, application number 202010292954X. , the entire contents of which are incorporated by reference into this disclosure.

Electronic map navigation has become a route search method that people mainly rely on when going out. In order to know the location of the desired store (destination), the only way is to rely on the floor plan at the entrance of the shopping mall. In many cases, it is not possible to clearly know the guidance route from the position to the destination, and therefore it often takes a lot of time and effort to search for the route to reach the destination.

In view of the above, an object of the present disclosure is to provide an indoor visual navigation method, apparatus, system and electronic equipment that can provide indoor navigation services to users and guide them to reach their destinations conveniently. That is.

To achieve the above objectives, the technical solutions used in the embodiments of the present disclosure are as follows.

In a first aspect, an embodiment of the present disclosure is a method of indoor visual navigation performed by a mobile device, wherein, after acquiring an indoor image to be positioned, the mobile device sets a camera pose when acquiring the indoor image. uploading the indoor image to the server for determination by the server; receiving a camera pose corresponding to the indoor image returned from the server; creating an AR coordinate system that matches the coordinate system; planning the shortest path in a pre-imported indoor topology map based on the destination information set by the user; displaying a preview image of on the screen of the mobile device, and superimposing a three-dimensional mark indicating a route traveling direction on the current preview image based on the AR coordinate system and the shortest path Provide a navigation method.

In one possible embodiment, the step of creating an AR coordinate system aligned with the world coordinate system based on the camera pose corresponding to the indoor image comprises: creating an initial AR coordinate system; adjusting the initial AR coordinate system based on the camera pose corresponding to the indoor image so that the coordinate system is aligned with the world coordinate system.

In one possible embodiment, the step of planning the shortest route in the pre-imported indoor topology map based on the destination information set by the user includes performing route planning based on the destination information set by the user. It involves using an algorithm to plan the shortest path in a pre-imported room topology map.

In one possible embodiment, the step of superimposing a three-dimensional mark indicating the direction of travel on the current preview image comprises detecting a ground plane in the current preview image; Determining the three-dimensional coordinates of the shortest route, generating a three-dimensional mark indicating the traveling direction of the route based on the determined three-dimensional coordinates, and drawing the three-dimensional mark on a ground plane in the current preview image. including.

In one possible embodiment, the method, upon receiving a current camera pose sent by the server in navigation, adjusts the current camera pose such that the corrected AR coordinate system remains aligned with the world coordinate system. Further comprising correcting the AR coordinate system based on the pose.

In a second aspect, an embodiment of the present disclosure is an indoor visual navigation method performed by a server, wherein upon receiving an indoor image of a positioning target uploaded by a mobile device, the mobile device collects the indoor image. determining a camera pose at a time, wherein the mobile device creates an AR coordinate system adapted to the world coordinate system based on the camera pose corresponding to the indoor image, and determines a route based on the AR coordinate system and the shortest path sending a camera pose corresponding to the indoor image to the mobile device such that a three-dimensional mark indicating the direction of travel is superimposed on a current preview image collected by the mobile device, wherein the shortest path is wherein the mobile device is planned in an indoor topology map based on the destination information set by the user.

In one possible embodiment, the step of determining a camera pose when the mobile device acquires the indoor image includes performing feature matching between the indoor image and a visual map in a pre-created visual map library, wherein the Obtaining a camera pose as the mobile device collects the indoor image, wherein the visual map is represented by a sparse point cloud model of the indoor scene.

In one possible embodiment, the step of performing feature matching between the room image and a visual map in a pre-created visual map library to obtain a camera pose when the mobile device collects the room image is deep hashed. calculating a global image descriptor for the indoor image using an algorithm; searching the visual map library for a plurality of keyframe images similar to the global image descriptor; obtaining respective keyframe information; dividing the plurality of keyframe images into a plurality of clusters according to the keyframe information; and traversing each of the clusters to obtain local feature points of the indoor image. computing local descriptors to match local feature points in the cluster; obtaining 3D map points corresponding to local feature points that are successfully matched; obtaining a camera pose corresponding to the indoor image if greater than a set number.

In one possible embodiment, the creation of the visual map library comprises the steps of acquiring a plurality of scene images acquired by a mobile device in an indoor scene; performing dimensional reconstruction to obtain a visual map library containing sparse point cloud models corresponding to the plurality of scene images.

In one possible embodiment, the method further comprises fitting the visual map to a previously imported interior floor plan.

In one possible embodiment, the method further comprises compressing the visual maps in the visual map library.

In one possible embodiment, compressing the visual maps in the visual map library includes encoding original features in the visual map; and removing the original features of .

In one possible embodiment, the method periodically acquires a current preview image acquired by the mobile device in navigation, and calculates a current camera pose when the mobile device acquires the current preview image. and transmitting the current camera pose to the mobile device such that the mobile device corrects the AR coordinate system based on the current camera pose.

In a third aspect, an embodiment of the present disclosure is an indoor visual navigation device disposed on a mobile device, wherein, after acquiring an indoor image to be positioned, a camera pose when the mobile device acquires the indoor image an image upload module configured to upload the indoor image to the server for determination by a server; and receiving a camera pose corresponding to the indoor image returned from the server and corresponding to the indoor image A coordinate system creation module configured to create an AR coordinate system that is aligned with the world coordinate system based on the camera pose and in the pre-imported indoor topology map based on the destination information set by the user a path planning module configured to plan a shortest path; and displaying a current preview image collected by the mobile device on an interface of the mobile device, based on the AR coordinate system and the shortest path. and a navigation display module configured to overlay a three-dimensional mark indicating a route travel direction on the current preview image.

In a fourth aspect, an embodiment of the present disclosure is an indoor visual navigation device located on a server, wherein upon receiving an indoor image of a positioning target uploaded by a mobile device, the mobile device collects the indoor image. a pose determination module configured to determine a camera pose when the mobile device creates an AR coordinate system adapted to a world coordinate system based on the camera pose corresponding to the indoor image; transmitting a camera pose corresponding to the room image to the mobile device such that a three-dimensional mark indicating the direction of travel based on the system and the shortest path is superimposed on a current preview image collected by the mobile device. wherein the shortest route is the one planned by the mobile device in an indoor topology map based on the destination information set by the user. I will provide a.

In a fifth aspect, an embodiment of the present disclosure includes a mobile device and a server communicatively coupled, the mobile device configured to perform the method of any one of the first aspects, The server provides an indoor visual navigation system configured to perform the method of any one of the second aspects.

In a sixth aspect, an embodiment of the present disclosure includes a processor and a storage device, wherein the storage device comprises the method of any one of the first aspect or the second aspect when run by the processor. An electronic device storing a computer program for performing the method of any one of claims 1 to 3 is provided.

In a seventh aspect, an embodiment of the present disclosure, when operated by a processor, performs the method steps of any one of the first aspect above or the method steps of any one of the second aspect above. A computer readable storage medium is provided that stores a computer program for execution.

The embodiments of the present disclosure provide an indoor visual navigation method, apparatus, system and electronic device, uploading the collected indoor images of the positioning target to the server by the mobile device, and posing the camera when the mobile device collects the indoor images. is determined by the server, and the camera pose corresponding to the indoor image is transmitted to the mobile device. Then, the mobile device creates an AR coordinate system that matches the world coordinate system based on the camera pose corresponding to the indoor image, and the user Based on the destination information set by , the shortest route is planned in the pre-imported indoor topology map, and finally the current preview image collected by the mobile device is displayed on the screen, and the AR coordinate system and the shortest route are plotted. By superimposing a three-dimensional mark indicating the direction of travel on the current preview image, indoor visual navigation is realized. The above embodiment according to the present embodiment guides the user to go to the destination by following the shortest route indoors using the AR method, thereby further improving the user experience.

Other features and advantages of embodiments of the disclosure may be set forth in the subsequent specification, or some of the features and advantages may be inferred or undoubtedly determined from the specification, or may be determined by practice of the disclosure. It can be ascertained by implementing the above techniques for examples.

In order to make the above objects, features and advantages of the present disclosure more comprehensible, preferred embodiments are described in detail below in conjunction with the accompanying drawings.

In order to describe the specific embodiments of the present disclosure or the technical solutions of the prior art more clearly, the drawings required for the description of the specific embodiments or the prior art are briefly described below. The drawings in the description are some embodiments of the present disclosure, and those skilled in the art can obtain other drawings based on these drawings without creative efforts.
1 shows a structural schematic diagram of an electronic device according to an embodiment of the present disclosure; FIG. 4 illustrates a flow chart of an example of an indoor visual navigation method according to an embodiment of the present disclosure; 4 shows a flowchart of another indoor visual navigation method according to an embodiment of the present disclosure; 4 shows a flowchart of another indoor visual navigation method according to an embodiment of the present disclosure; 1 shows a structural block diagram of an indoor visual navigation device according to an embodiment of the present disclosure; FIG. FIG. 3 shows a structural block diagram of another indoor visual navigation device according to an embodiment of the present disclosure;

In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the following describes the technical solutions of the present disclosure with reference to the drawings. Each described embodiment is only a part of the possible embodiments, but not all of them.

For the problem that users cannot use mobile terminals such as mobile phones for indoor navigation, the embodiments of the present disclosure provide an indoor visual navigation method, apparatus, system and electronic equipment, which are The embodiments of the present disclosure, which can be applied to any case requiring indoor navigation, are described in detail below.

First, referring to FIG. 1, an example of an electronic device 100 that implements an indoor visual navigation method, apparatus, system, and electronic device according to embodiments of the present disclosure is shown.

As a structural schematic diagram of an electronic device shown in FIG. 1, an electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108 and an image acquisition device 110, These components are interconnected via bus system 112 and/or other forms of connection mechanisms (not shown). It should be noted that the components and structures of the electronic device 100 shown in FIG. 1 are exemplary and not limiting, and the electronic device may have other components and structures as needed. .

The processor 102 may be implemented in the form of hardware of at least one of a digital signal processor (DSP), a field programmable gate array (FPGA), and a programmable logic array (PLA), and the processor 102 is a central It may be one or more combinations of a processing unit (CPU), a figure processing unit (GPU), or other form of processing unit having data processing capability and/or instruction execution capability to perform the desired functions. Other components of the electronic device 100 can be controlled to run.

The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile and/or non-volatile memory. . The volatile memory may include, for example, random access memory (RAM) and/or cache. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 executes the program instructions to cause the processor-implemented embodiments of the present disclosure described below. ) and/or other desired functionality. Various application programs and data, such as various data used and/or generated by the application programs, may be stored in the computer-readable storage medium.

The input device 106 may be a device for user input of commands and includes one or more of a keyboard, mouse, microphone, touch screen, and the like.

The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a monitor, speakers, and the like.

The image capture device 110 may capture images desired by a user (eg, photos, videos, etc.) and store the captured images in the storage device 104 for use by other components.

Illustratively, exemplary electronic devices of the indoor visual navigation methods, devices, systems and electronic devices for implementing embodiments of the present disclosure are smart terminals such as smart phones, tablets, wearable electronic devices, computers, servers, etc. may be implemented as

Embodiments of the present disclosure may provide a mobile device-side indoor visual navigation method, which may be performed by a mobile device, such as a smartphone, tablet, or wearable electronic device, such as an indoor navigation device illustrated in FIG. Referring to the visual navigation method flowchart, the method mainly includes steps S202 to S208 as follows.

Step S202, after collecting the indoor image to be positioned, upload the indoor image to the server so that the camera pose when the mobile device collects the indoor image is determined by the server.

In the following, a mobile device will be exemplified as a mobile phone, and the user may not know his/her current position when in an indoor scene, and may not know how to get to the destination from the current position. First, the mobile phone captures the indoor image of the current scene, and then the indoor image is uploaded to the server, and the server can perform visual positioning based on the indoor image.

Step S204: Receive the camera pose corresponding to the indoor image returned by the server, and create an AR coordinate system adapted to the world coordinate system based on the camera pose corresponding to the indoor image.

AR (Augmented Reality) technology is a technology that fuses virtual information and the real world. The virtual graphics and real scene images can be displayed with the effect of being on the same screen or in the same space, allowing the user to experience a scene that combines virtual and reality. In one possible embodiment, realized by an AR navigation scheme, the virtual figures need to be displayed in a real scene, so an AR coordinate system is created and adapted to the room image so that the AR coordinate system aligns with the world coordinate system. We need to adjust the AR coordinate system based on the desired camera pose. Here, creation of the AR coordinate system can be realized using a conventional SLAM algorithm (Simultaneous Localization and Mapping, simultaneous execution of self-location estimation and environment map creation), and will not be described in detail here.

Step S206: Plan the shortest route in the pre-imported indoor topology map according to the destination information set by the user.

The user can input the name of the destination to the mobile phone APP, or click the destination on the interior floor plan displayed on the mobile phone APP, and is not limited here. When the mobile phone acquires the destination information set by the user, it can plan the route in the pre-imported indoor topology map. In the indoor topology map, each store may be a node in the map, and the indoor route may be an edge in the map. In one possible embodiment, for areas that need to provide indoor navigation services, such as shopping malls, libraries, museums, etc., the server may be pre-provided with an indoor topology map, or the server may generate a map based on the indoor floor plan. can be converted directly.

Step S208, display the current preview image collected by the mobile device on the screen of the mobile device, and superimpose the 3D mark indicating the route traveling direction on the current preview image according to the AR coordinate system and the shortest path.

The camera of the mobile device is first in a shooting state, collects images continuously, and displays the collected images on the screen of the mobile terminal, which means that the camera of the mobile device is in image preview mode. may be considered. When a user carries out navigation with a mobile phone, the camera of the mobile phone is in preview mode, and the user can see, through the mobile phone screen, that the 3D mark indicating the direction of travel is currently in the indoor scene (for example, the ground). (marked with an arrow) can be visually recognized.

According to the indoor visual navigation method according to the present embodiment, the mobile device can guide the user to follow the shortest route to the destination indoors by the method of AR, so as to improve the user experience.

When creating an AR coordinate system that matches the world coordinate system based on the camera pose corresponding to the indoor image, first create the initial AR coordinate system, and then adjust the AR coordinate system to the world coordinate system. , the initial AR coordinate system may be adjusted based on the camera pose corresponding to the indoor image.

The mobile phone may be provided with an AR system, and the AR system may be Visual-Inertial Odometry (VIO), usually with loopback detection function, where the AR coordinate system is the VIO coordinate may be called a system. The AR system can be AR Kit attached to iOS system, or AR Core attached to Android, or any third-party system capable of realizing mobile device navigation function, and is not limited here.

The initially created AR coordinate system usually has the first frame image acquired as the coordinate origin. It is necessary to adjust the initial AR coordinate system using the camera pose corresponding to the indoor image so that the virtual image drawn based on the AR coordinate system can be well blended with the real scene image in the world coordinate system. There is This allows the mobile phone to smoothly transform the planned route into the world coordinate system and well combine the virtual graphics with the real scene image.

In one possible embodiment, the examples of the present disclosure provide a specific embodiment of planning the shortest route in a pre-imported indoor topology map based on the destination information set by the user. That is, based on the destination information set by the user, a route planning algorithm is used to plan the shortest route in the pre-imported indoor topology map. The path planning algorithm may be the A* algorithm, but of course other path planning algorithms are not limiting here.

In order to provide the user with a clear navigation direction, in this embodiment, when performing the above step S208, the following steps may be referred to for implementation. (1) Detect the ground plane in the current preview image. (2) Determine the three-dimensional coordinates of the shortest route in the AR coordinate system, and generate a three-dimensional mark indicating the traveling direction of the route based on the determined three-dimensional coordinates. (3) Draw a 3D mark on the ground plane in the current preview image. For example, the three-dimensional mark may be a virtual arrow, a dotted line path mark, or the like. The user can finally reach the destination by the shortest route by following the direction indicated by the three-dimensional mark.

In AR navigation, considering factors such as initial visual positioning pose errors and drift of the AR system itself, the accuracy of the navigation path may degrade over time, for example, when the navigation path crosses buildings. There is Therefore, in order to ensure accurate navigation throughout the journey, the mobile device in this embodiment, upon receiving the current camera pose sent by the server in navigation, will keep the corrected AR coordinate system aligned with the world coordinate system. Correct the AR coordinate system based on the current camera pose, such as That is, the server periodically obtains the current preview image collected by the mobile device in navigation, determines the current camera pose when the mobile device collects the current preview image, and sets the current camera pose to the mobile device. , and then the mobile device corrects the AR coordinate system based on the current camera pose, which improves navigation accuracy.

In this embodiment, a server-side indoor visual navigation method is also provided, which may be performed by, for example, a cloud server. Referring to the indoor visual navigation method flowchart shown in FIG. It mainly includes the following steps S302 to S304.

Step S302, upon receiving the indoor image for positioning uploaded by the mobile device, determining a camera pose when the mobile device collects the indoor image. Here, the camera pose may include XY coordinates and camera orientation.

Step S304, the mobile device creates an AR coordinate system according to the world coordinate system according to the camera pose corresponding to the indoor image, and creates a three-dimensional mark indicating the traveling direction of the route on the mobile device based on the AR coordinate system and the shortest route. to the mobile device, where the shortest path is the camera pose corresponding to the indoor image to be superimposed on the current preview image collected by the mobile device based on the destination information set by the user. It is planned in the topology map.

According to the above indoor visual navigation method according to the present embodiment, the server is in charge of the computationally intensive steps required during navigation, such as camera pose calculation, and then the mobile device is indoors by means of AR. , the user can be easily and efficiently guided to the destination along the shortest route, and the user experience can be further improved.

In this example, all time and space consuming computations are performed on the server side, e.g. camera poses are determined by the server when the mobile device collects room images, and in one possible embodiment the server: Feature matching may be performed between the room image and a visual map in a pre-created visual map library to obtain a camera pose when the mobile device collects the room image, where the visual map is a sparse interior scene. It is represented by a point cloud model and may be understood as a mass of visual features of the scene. In one possible embodiment, we solve the Perspective-n-Point (PNP) problem according to the 3D-2D matching relation, i.e., how to move point pairs from 3D to 2D, which allows us to solve the motion of feature point pairs. The camera pose can be solved based on

This example provides one possible embodiment of performing feature matching between an indoor image and a visual map in a pre-created visual map library to obtain camera poses as the mobile device collects the indoor image. , mainly includes two stages of coarse positioning and fine positioning, each of which is described as follows.

In the coarse positioning stage, a deep hash algorithm is used to compute the global image descriptor of the indoor image, search the visual map library for the k keyframe images that are most similar to the global image descriptor, and extract the keyframe Having obtained the ID, then based on the keyframe ID, retrieve the stored keyframe information including the pose of the keyframe, the local feature points, the local descriptors and the coordinates in the world coordinate system of the corresponding map points. These k keyframes are clustered according to the poses of the keyframes, and keyframes with similar positions are put into the same cluster. For each cluster, the center of the cluster provides initial coarse positioning results for subsequent fine positioning.

In the fine positioning stage, first, each of the clusters is traversed, local feature points are extracted from the room image, local descriptors are computed and matched with the local features of all keyframes in the cluster, and then matching and if the number of 3D-2D point pairs with successful matching is greater than a preset number (e.g., greater than 5), solve the PNP problem, A camera pose corresponding to the indoor image can be obtained. Using the pose obtained by PNP as an initial value, we can further construct a Bundle Adjustment graph optimization problem, which optimizes the pose corresponding to the room image and minimizes the reprojection error. After the pose is optimized, we remove the edges with still large reprojection errors, use the remaining edges to reconstruct the Bundle Adjustment graph optimization problem, and finally get the correct pose corresponding to the room image. . In this process, if the number of 3D-2D point pairs is too small or the reprojection error is too large after optimization, the keyframes in the current cluster are considered to be error matching, This cluster is abandoned. If the reprojection error is small after optimization, we consider the pose solution to be correct and output the result directly, avoiding entering the cycle for the next cluster.

The server can obtain an accurate camera pose corresponding to the indoor image by performing the above two stages of coarse positioning and fine positioning in this order. Of course, the above is just one pose determination method according to this embodiment, and may be implemented in any other manner for determining a camera pose, and is not limited here.

In one possible embodiment, the server also aligns the visual map with the pre-imported room floor plan so that the indoor navigation mobile device can successfully transform the planned route into the world coordinate system. Here, the interior floor plan may be understood as a building structure diagram, and for example, the interior floor plan may be uploaded in advance to a server by a shopping mall or the like.

The server may pre-build a visual map library, and creating a visual map library includes the following steps. (1) Acquire a plurality of scene images collected by a mobile device in an indoor scene. In one possible embodiment, a large number of images of each scene in the room may be pre-collected in order to build an accurate sparse point cloud model. (2) 3D reconstruction is performed on multiple scene images based on the SFM (Structure From Motion) algorithm to obtain a visual map library containing sparse point cloud models corresponding to the multiple scene images. SFM mapping may be implemented by open source algorithms such as COLMAP, Theia, VisualSfM, OpenMVG, etc., and is not limited here.

In one possible embodiment, in order to save storage space on the server's magnetic disk, the visual maps in the visual map library may be compressed, e.g. , and storing only the encoded result in the visual map and not the original features, the size of the map can be greatly reduced. The server may decode the encoded visual features and perform matching and pose estimation using the decoded features in performing visual positioning using the visual database.

The above scheme allows the server to take charge of the computationally intensive visual map library construction and camera pose determination, which reduces the hardware requirements of the mobile device and allows the mobile device to follow the calculation results of the server. Based on this, navigation services can be provided to users more efficiently.

In one possible embodiment, in AR navigation, the accuracy of the navigation path may degrade over time, considering factors such as initial visual positioning pose errors and drafts of the AR system itself, and the server also periodically obtains the current preview image acquired by the mobile device in navigation, determines the current camera pose when the mobile device acquires the current preview image, and determines the current camera pose when the mobile device acquires the current preview image. The current camera pose may be sent to the mobile device so that it can correct the AR coordinate system.

The embodiments of the present disclosure also provide one possible embodiment of the mobile device-side and server-side room visual navigation method, specifically refer to the flowchart of the room visual navigation method shown in FIG. and includes the following steps:
Step S410: Collect a plurality of indoor images.
Step S412: Perform SFM mapping based on the indoor image to generate a visual map database.
Step S414: Align the visual map in the visual map database with the interior floor plan. Here, the interior floor plan may be referred to as a building structure drawing.
Step S420: Receive the indoor image for positioning uploaded by the mobile phone.
Step S422: Extract the image features of the indoor image to be positioned, match the extracted image features with the visual map, and estimate the camera pose when the mobile phone collects the indoor image.
Step S424: Return the estimated camera pose to the mobile phone.
Step S430: Create an AR coordinate system.
Step S432: Align the AR coordinate system with the camera pose when the mobile phone collects the indoor image.
Step S434: Receive the destination information set by the user and plan the shortest route in the indoor topology map.
Step S436: Detect the ground plane.
Step S438: Transform the shortest path into three-dimensional coordinates in the AR coordinate system, and draw the path on the ground plane with arrows.

For the detailed operation of the above steps, please refer to the contents of the indoor visual navigation method by the mobile device side and the indoor visual navigation method by the server side, and will not be described in detail herein.

Here, steps S410 to S414 may be collectively referred to as a visual map construction operation, and steps S420 to S424 may be collectively referred to as a cloud side visual positioning operation. may all be performed by a server, and steps S430 to S438 may be collectively referred to as a mobile-side AR navigation operation, and may be performed by a mobile device such as a mobile phone.

With the above indoor visual navigation method according to the present embodiment, the user can use the mobile phone to locate himself/herself by photographing the surrounding environment anytime and anywhere, and after selecting the destination, the mobile phone will The optimal route planned by the route selection algorithm can be confirmed on the screen of the mobile phone, and the destination can be reached by following the route. In addition, in the above indoor visual navigation method, time- and space-consuming computations are performed on the server side, thereby enabling real-time positioning and navigation on mobile devices.

An embodiment of the present disclosure further provides an indoor visual navigation device located on the mobile device side, as shown in the structural block diagram of the indoor visual navigation device shown in FIG. 5, the indoor visual navigation device includes:
an image uploading module 502 configured to, upon acquiring the indoor images to be positioned, upload the indoor images to the server for the server to determine the camera pose when the mobile device acquires the indoor images;
a coordinate system creation module 504 configured to receive the camera pose corresponding to the indoor image returned by the server and create an AR coordinate system aligned with the world coordinate system based on the camera pose corresponding to the indoor image; ,
a route planning module 506 configured to plan the shortest route in the pre-imported indoor topology map based on the destination information set by the user;
configured to display a current preview image collected by the mobile device on a screen of the mobile device, and superimpose a three-dimensional mark indicating a route traveling direction on the current preview image based on the AR coordinate system and the shortest path; and a navigation display module 508 .

According to the indoor visual navigation device according to the present embodiment, the mobile device can guide the user to the destination by simply and efficiently following the shortest route indoors using the AR method, thereby further improving the user experience. can be done.

In one possible embodiment, the coordinate system creation module 504 creates an initial AR coordinate system and calculates the initial AR coordinates based on the camera pose corresponding to the indoor image so that the AR coordinate system aligns with the world coordinate system. configured to regulate the system.

In one possible embodiment, the route planning module 506 is configured to utilize a route planning algorithm to plan the shortest route in a pre-imported indoor topology map based on the destination information.

In one possible embodiment, the navigation display module 508 detects the ground plane in the current preview image, determines the 3D coordinates of the shortest route in the AR coordinate system, and performs route progression based on the determined 3D coordinates. It is configured to generate a 3D mark indicating direction and to draw the 3D mark on the ground plane in the current preview image.

In one possible embodiment, once the device receives the current camera pose sent by the server in navigation, the device adjusts based on the current camera pose so that the corrected AR coordinate system remains aligned with the world coordinate system. further including a coordinate system correction module configured to correct the AR coordinate system using the coordinate system.

The apparatus according to this embodiment, its implementation principle and the technical effects produced are the same as those of the previous embodiments, and for the sake of simplicity of explanation, those not described in the embodiment of the apparatus are: Please refer to the corresponding content in the foregoing method embodiments.

An embodiment of the present disclosure further provides a server-side indoor visual navigation device, as shown in the structural block diagram of the indoor visual navigation device shown in FIG. 6, the indoor visual navigation device includes:
a pose determination module 602 configured to determine a camera pose when the mobile device collects the indoor image upon receiving the indoor image of the positioning target uploaded by the mobile device;
Based on the camera pose corresponding to the indoor image, the mobile device creates an AR coordinate system that matches the world coordinate system, and the mobile device collects 3D marks indicating the direction of travel based on the AR coordinate system and the shortest route. a device navigation module 604 configured to transmit a camera pose corresponding to the room image to the mobile device to overlay the displayed current preview image, wherein the shortest path to the mobile device is set by the user. and an equipment navigation module 604, which is planned in the indoor topology map based on the destination information.

In the above indoor visual navigation device according to the present embodiment, the server takes charge of the steps necessary for navigation, such as camera pose calculation, which require a large amount of calculation. It is possible to simply and efficiently guide the user to go to the ground and further improve the user experience.

In one embodiment, the pose determination module 602 is configured to perform feature matching between the room image and a visual map in a pre-created visual map library to obtain the camera pose when the mobile device collects the room image. , the visual map is represented by a sparse point cloud model of the indoor scene.

In one embodiment, the above apparatus obtains a plurality of scene images collected in an indoor scene by a mobile device, performs 3D reconstruction on the plurality of scene images based on an SFM algorithm, and obtains a plurality of scene images. Further includes a map creation module configured to obtain a visual map library containing sparse point cloud models corresponding to the scene image.

In one embodiment, the apparatus further includes a matching module configured to match the visual map with the previously imported interior floor plan.

In one embodiment, the apparatus is configured to periodically acquire a current preview image acquired by the mobile device in navigation and to determine a current camera pose when the mobile device acquires the current preview image. and a correction module configured to transmit the current camera pose to the mobile device such that the mobile device corrects the AR coordinate system based on the current camera pose.

The apparatus according to the embodiment of the present disclosure, its implementation principle, and the technical effects generated are the same as those of the above-described embodiments. Please refer to the corresponding content in the embodiment of the indoor visual navigation method by the device side.

Embodiments of the present disclosure further provide an indoor visual navigation system including a mobile device and a server.

An embodiment of the present disclosure includes a processor and a storage device, and the storage device stores a computer program for executing a mobile device-side indoor visual navigation method or a server-side indoor visual navigation method when run by the processor. It further provides an electronic device with a

Embodiments of the present disclosure further provide a computer-readable storage medium storing a computer program that, when run by a processor, performs a mobile-side indoor visual navigation method or a server-side indoor visual navigation method.

As will be apparent to those skilled in the art, for the convenience and simplicity of explanation, the operating steps of the above-described system can be specifically referred to the corresponding steps in the preceding embodiments, and will not be described in detail herein. No explanation.

An indoor visual navigation method, apparatus, system, and computer program product for an electronic device according to embodiments of the present disclosure include a computer readable storage medium having program code stored thereon, instructions contained in the program code for executing the method. , the specific implementation can refer to the foregoing content of the embodiments of the present disclosure, which will not be described in detail herein.

Also, in the description of the embodiments of the present disclosure, the terms “attach”, “connect”, and “connect” should be understood broadly, unless otherwise clearly defined or limited. connection, integral connection, mechanical connection, electrical connection, direct connection, indirect connection via an intermediate member, and internal communication between the two elements. . A person of ordinary skill in the art can understand the specific meaning of the above terms in this disclosure depending on the specific situation.

The functionality may be implemented in the form of software functional units and stored on a computer-readable storage medium when sold or used as an independent product. Based on such knowledge, the gist of the technical solution of the present disclosure, the part contributing to the prior art, or part of the technical solution may be embodied in the form of a software product, and the computer software product is stored in a storage medium, and can cause a single computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method described in each embodiment of the present disclosure; Contains some instructions. The aforementioned storage media include USB memory, removable hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, compact disk, etc., which can store program code. It may be a medium.

In the description of the present disclosure, directions indicated by terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", and "outside" or are based on the orientation or positional relationships shown in the drawings, and are merely for simplicity and simplicity of description of the present disclosure, and such devices or elements do not necessarily have a particular orientation or have a particular orientation. It is not intended or implied to be configured or operated in any orientation, and as such should not be construed as limiting the present disclosure. Furthermore, the terms "first," "second," and "third" are for descriptive purposes only and should not be understood as indicating or implying relative importance.

It should be noted that the above examples are only preferred embodiments of the present disclosure, and are intended to describe the technical solutions of the present disclosure and are not intended to be limiting, and the patent scope of the present disclosure is limited to these Although the present disclosure has been described in detail with reference to the foregoing examples, without being limited to You can modify the technical solutions described in the above-described embodiments, easily conceive of changes, or make equivalent substitutions for part of the technical features, and these modifications and changes or replacement, the gist of the corresponding technical solutions does not depart from the spirit and scope of the technical solutions in the embodiments of the present disclosure, and they are all covered by the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should conform to the protection scope of the claims.

<Industrial applicability>
In the embodiment of the present disclosure, the mobile device uploads the collected indoor image of the positioning target to the server, the server determines the camera pose when the mobile device collects the indoor image, and the camera pose corresponding to the indoor image is moved. After being sent to the device, the mobile device creates an AR coordinate system aligned with the world coordinate system based on the camera pose corresponding to the indoor image, and based on the destination information set by the user, the pre-imported The shortest path is planned in the indoor topology map, and finally the current preview image collected by the mobile device is displayed on the screen, and the 3D mark indicating the path traveling direction is displayed on the current preview image based on the AR coordinate system and the shortest path. Provide an indoor visual navigation method, apparatus, system and electronic equipment for realizing indoor visual navigation by overlaying. In this way, the user can be guided to the destination by following the shortest route indoors using the AR method, and the user experience can be further improved.

Claims (18)

An indoor visual navigation method performed by a mobile device, comprising:
After collecting the indoor images to be positioned, uploading the indoor images to the server for determination by the server of a camera pose when the mobile device collects the indoor images;
a step of receiving a camera pose corresponding to the indoor image returned from the server and creating an AR coordinate system adapted to the world coordinate system based on the camera pose corresponding to the indoor image;
planning the shortest path in the pre-imported indoor topology map based on the destination information set by the user;
displaying a current preview image collected by the mobile device on a screen of the mobile device, and superimposing a three-dimensional mark indicating a route traveling direction on the current preview image based on the AR coordinate system and the shortest path; A room visual navigation method, comprising: The step of creating an AR coordinate system adapted to the world coordinate system based on the camera pose corresponding to the indoor image,
creating an initial AR coordinate system;
adjusting the initial AR coordinate system based on a camera pose corresponding to the indoor image so that the AR coordinate system aligns with the world coordinate system. planning the shortest path in the pre-imported indoor topology map based on the destination information set by the user;
3. A method according to claim 1 or 2, comprising using a route planning algorithm to plan the shortest route in a pre-imported indoor topology map based on the destination information set by the user. The step of superimposing a three-dimensional mark indicating the traveling direction of the route on the current preview image,
detecting a ground plane in the current preview image;
determining the three-dimensional coordinates of the shortest route in the AR coordinate system, and generating a three-dimensional mark indicating the traveling direction of the route based on the determined three-dimensional coordinates;
and drawing the three-dimensional mark on the ground plane in the current preview image. Upon receiving a current camera pose sent by the server in navigation, correct the AR coordinate system based on the current camera pose such that the corrected AR coordinate system remains aligned with the world coordinate system. The method of any one of claims 1-4, further comprising steps. An indoor visual navigation method executed by a server, comprising:
Upon receiving an indoor image of a positioning target uploaded by a mobile device, determining a camera pose when the mobile device collects the indoor image;
The mobile device creates an AR coordinate system that matches the world coordinate system based on the camera pose corresponding to the indoor image, and moves the three-dimensional mark indicating the traveling direction of the route based on the AR coordinate system and the shortest route. sending a camera pose corresponding to the room image to the mobile device so as to be overlaid on a current preview image collected by the device, wherein the shortest path is a destination for which the mobile device has been set by a user; planning in an indoor topology map based on ground information. Determining a camera pose when the mobile device acquires the room image comprises:
performing feature matching between the indoor image and a visual map in a pre-created visual map library to obtain a camera pose when the mobile device collects the indoor image, wherein the visual map is a sparse indoor scene; 7. The method of claim 6, comprising steps represented by a point cloud model. performing feature matching between the room image and a visual map in a pre-created visual map library to obtain a camera pose when the mobile device collects the room image,
calculating a global image descriptor for the indoor image using a deep hashing algorithm;
searching the visual map library for a plurality of keyframe images that are similar to the global image descriptor;
obtaining keyframe information for each of the keyframe images;
dividing the plurality of keyframe images into a plurality of clusters according to the keyframe information;
traversing each of the clusters to obtain local feature points of the room image;
computing local descriptors to match local feature points within the cluster;
obtaining 3D map points corresponding to successfully matched local feature points;
and obtaining a camera pose corresponding to the room image if the number of 3D map points is greater than a preset number. The creation of the visual map library includes:
acquiring a plurality of scene images acquired by a mobile device in an indoor scene;
performing 3D reconstruction on the plurality of scene images based on an SFM algorithm to obtain a visual map library containing sparse point cloud models corresponding to the plurality of scene images. Or the method according to 8. A method according to any one of claims 7 to 9, further comprising fitting said visual map to a previously imported interior floor plan. A method according to any one of claims 7 to 10, further comprising compressing the visual maps in said visual map library. Compressing the visual maps in the visual map library comprises:
encoding original features in the visual map;
preserving the encoded original features and removing the original features before encoding. periodically acquiring a current preview image acquired by the mobile device in navigation and determining a current camera pose when the mobile device acquires the current preview image;
transmitting the current camera pose to the mobile device such that the mobile device corrects the AR coordinate system based on the current camera pose. The method described in section. An indoor visual navigation device arranged on a mobile device side,
an image upload module configured to, upon acquiring an indoor image to be positioned, upload the indoor image to the server for determination by the server of a camera pose when the mobile device acquires the indoor image;
Coordinate system creation configured to receive a camera pose corresponding to the indoor image returned from the server and create an AR coordinate system adapted to the world coordinate system based on the camera pose corresponding to the indoor image a module;
a route planning module configured to plan the shortest route in the pre-imported indoor topology map based on the destination information set by the user;
displaying a current preview image collected by the mobile device on a screen of the mobile device, and superimposing a three-dimensional mark indicating a route traveling direction on the current preview image based on the AR coordinate system and the shortest path; a navigation display module configured to mate with the indoor visual navigation device. An indoor visual navigation device located on the server side,
a pose determination module configured to, upon receiving an indoor image of a positioning target uploaded by a mobile device, determine a camera pose when the mobile device collects the indoor image;
The mobile device creates an AR coordinate system that matches the world coordinate system based on the camera pose corresponding to the indoor image, and moves the three-dimensional mark indicating the traveling direction of the route based on the AR coordinate system and the shortest route. a device navigation module configured to transmit a camera pose corresponding to the room image to the mobile device overlaid on a current preview image collected by the device, wherein the shortest path is the mobile device is planned in the indoor topology map based on the destination information set by the user. An indoor visual navigation system including a mobile device and a server communicatively connected, comprising:
The mobile device is configured to perform the method according to any one of claims 1-5 and the server is adapted to perform the method according to any one of claims 6-13. indoor visual navigation system. including a processor and storage device;
The storage device stores a computer program that, when run by the processor, executes the method according to any one of claims 1 to 5 or the method according to any one of claims 6 to 13. electronic equipment. A computer program is stored which, when run by a processor, performs the steps of the method according to any one of claims 1-5 or the steps of the method according to any one of claims 6-13. computer-readable storage medium.

Images