WO2022170736A1

WO2022170736A1 - Navigation prompt method and apparatus, and electronic device, computer-readable storage medium, computer program and program product

Info

Publication number: WO2022170736A1
Application number: PCT/CN2021/106909
Authority: WO
Inventors: 陈思平; 张国伟
Original assignee: 深圳市慧鲤科技有限公司
Priority date: 2021-02-09
Filing date: 2021-07-16
Publication date: 2022-08-18
Also published as: CN112861725A

Abstract

A navigation prompt method and apparatus, and an electronic device, a computer-readable storage medium, a computer program and a program product. The navigation prompt method comprises: acquiring a real scene image captured by an augmented reality (AR) device (S101); on the basis of the real scene image, determining whether the AR device is located within a geographical position range corresponding to a target region (S102); when the AR device is within the geographical position range corresponding to the target region, detecting whether a target object is present in the real scene image (S103); if a target object is present in the real scene image, determining whether there is a risk of a collision occurring between the AR device and the target object (S104); and when there is a risk of a collision occurring between the AR device and the target object, providing an early warning prompt for a user by means of the AR device (S105).

Description

Navigation prompt method, apparatus, electronic device, computer readable storage medium, computer program and program product

CROSS-REFERENCE TO RELATED APPLICATIONS

This disclosure claims the priority of the Chinese patent application with the application number of 202110176459.7 and the application title of "A Navigation Prompting Method, Device, Electronic Device and Storage Medium" filed with the China Patent Office on February 9, 2021. The Chinese patent application The entire contents of is hereby incorporated by reference into this disclosure in its entirety.

technical field

The present disclosure relates to the technical field of navigation, and in particular, to a navigation prompting method, apparatus, electronic device, computer-readable storage medium, computer program, and program product.

Background technique

With the development of computer technology, more and more applications for navigation have appeared, which brings convenience to people's travel life.

While the application for navigation brings the convenience of navigation to people's travel, how to improve the safety of users when using navigation is an urgent problem to be solved.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide at least a navigation prompting method, apparatus, electronic device, computer-readable storage medium, computer program, and program product.

In a first aspect, an embodiment of the present disclosure provides a navigation prompt method, including:

Obtain images of real scenes captured by Augmented Reality (AR) devices;

Based on the real scene image, determine whether the AR device is located within the geographic location range corresponding to the target area;

In the case that the AR device is located within the geographic location range corresponding to the target area, detecting whether there is a target object in the real scene image;

In the case that there is a target object in the real scene image, determine whether there is a risk of collision between the AR device and the target object;

In the case that there is a risk of collision between the AR device and the target object, an early warning prompt is given to the user through the AR device.

In the embodiment of the present disclosure, it is proposed that when the AR device is located in the target area, target detection can be performed on the real scene image collected by the AR device, there is a target object in the real scene image, and it is estimated that the AR device and the target object collide In the case of high risk, the AR device can be used to warn the user to avoid the safety accident caused by the collision between the user and the target object during the navigation process, thereby improving the user's travel safety.

In a possible implementation manner, the determining whether the AR device is located within a geographic location range corresponding to the target area based on the real scene image includes:

Based on the real scene image and the preset three-dimensional scene map, determine the first pose information of the AR device in the world coordinate system; the three-dimensional scene map is a map representing the real scene to which the target area belongs;

Based on the first pose information and the geographic location range of each area associated with the three-dimensional scene map in the world coordinate system, it is determined whether the AR device is located within the geographic location range corresponding to the target area.

In the embodiment of the present disclosure, it is proposed that the first pose information of the AR device can be determined according to the real scene image and the preset three-dimensional scene map. The first pose information of the device and the geographic location range corresponding to each area can quickly determine whether the AR device is located in the target area to be alerted, such as on the road where vehicles pass, so as to ensure the user's safe travel.

In a possible implementation manner, in the case where a target object exists in the real scene image, judging whether there is a risk of collision between the AR device and the target object includes:

Based on the real scene image, determine the relative pose information between the target object and the AR device;

Based on the relative pose information, it is determined whether there is a risk of collision between the AR device and the target object.

In the embodiment of the present disclosure, it is proposed that the relative pose information between the target object and the AR device, such as relative distance, orientation relationship, etc., can be determined based on the real scene image, so that the relative pose information can be used to estimate the relationship between the AR device and the AR device. Whether there is a risk of collision between target objects provides a guarantee for users to travel safely.

In a possible implementation manner, the target object includes a static obstacle, and the determining whether there is a risk of collision between the AR device and the target object based on the relative pose information includes:

Based on the relative pose information, determine whether the distance between the AR device and the static obstacle is less than a first preset distance;

In the case where the distance between the AR device and the static obstacle is less than a first preset distance, determine whether the AR device is facing the static obstacle;

In the case where the AR device faces the static obstacle, it is determined that there is a risk of collision between the AR device and the static obstacle.

In the embodiment of the present disclosure, a method for determining whether there is a risk of collision against a static obstacle is provided. For example, if the target object is a railing, the distance between the AR device and the railing can be relatively short, and the AR device is facing the railing. Next, it is determined that there is a risk of a collision due to the proximity of the AR device to the railing.

In a possible implementation manner, the target object includes a dynamic obstacle, and the determining whether there is a risk of collision between the AR device and the target object based on the relative pose information includes:

Determine relative motion information between the AR device and the dynamic obstacle based on a plurality of real scene images;

Based on the relative pose information and the relative motion information, it is determined whether there is a risk of collision between the AR device and the dynamic obstacle.

In the embodiment of the present disclosure, a method for determining whether there is a risk of collision with respect to a dynamic obstacle is provided, which can determine the relative motion information between the target object and the AR device based on multiple real scene images, which can include the relative motion direction , relative motion speed, etc., in this way, the relative pose information and relative motion information can be combined to estimate whether there is a risk of collision between the AR device and the target object, and provide a guarantee for the user to travel safely.

In a possible implementation manner, the determining whether there is a risk of collision between the AR device and the dynamic obstacle based on the relative pose information and the relative motion information includes:

Based on the relative pose information, determine whether the distance between the AR device and the dynamic obstacle is less than a second preset distance;

In the case that the distance between the AR device and the dynamic obstacle is less than the second preset distance, based on the relative motion information, it is determined that the driving direction of the dynamic obstacle relative to the AR device is different from that of the dynamic obstacle. Whether the included angle between the directions of the dynamic obstacles toward the AR device is less than the set angle threshold;

In the case that the included angle is smaller than the set angle threshold, it is determined that there is a risk of collision between the AR device and the dynamic obstacle.

In the embodiment of the present disclosure, a method is provided for determining whether there is a risk of collision in combination with the motion state of the AR device and/or the dynamic obstacle, so that the distance between the AR device and the dynamic obstacle is less than a second preset distance, And when the dynamic obstacle and the AR device are close to each other, it can accurately predict the risk of collision between the AR device and the dynamic obstacle.

In a possible implementation manner, the determining relative pose information between the target object and the AR device based on the real scene image includes:

According to the real scene image, the detection area corresponding to the target object in the real scene image, and the preset three-dimensional scene map, determine the second pose information of the target object in the world coordinate system; the The three-dimensional scene map is a map representing the real scene to which the target area belongs;

Positioning the AR device to obtain the first pose information of the AR device in the world coordinate system;

Determine relative pose information between the target object and the AR device according to the first pose information and the second pose information.

In the embodiments of the present disclosure, it is proposed that by determining the respective pose information of the target object and the AR device in the world coordinate system, the relative pose information between the target object and the AR device can be quickly and accurately determined.

In a possible implementation manner, the performing an early warning prompt to the user through the AR device includes:

An early warning prompt is given to the user through the AR device in at least one of voice form, text form, animation form, warning sign, and flashing form.

In the embodiment of the present disclosure, it is proposed that the user can be warned in various ways, so as to improve the travel safety of the user.

In a second aspect, an embodiment of the present disclosure provides a navigation prompting device, including:

The image acquisition part is configured to acquire the real scene image captured by the augmented reality AR device;

a first judging part, configured to judge whether the AR device is located within the geographic location range corresponding to the target area based on the real scene image;

a target detection part, configured to detect whether a target object exists in the real scene image when the AR device is located within the geographic location range corresponding to the target area;

The second judgment part is configured to judge whether there is a risk of collision between the AR device and the target object when there is a target object in the real scene image;

The pre-warning and prompting part is configured to provide a pre-warning prompt to the user through the AR device when there is a risk of collision between the AR device and the target object.

In a third aspect, embodiments of the present disclosure provide an electronic device, including: a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the processing The processor and the memory communicate through a bus, and the machine-readable instructions execute the navigation prompt method according to the first aspect when the machine-readable instructions are executed by the processor.

In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the navigation prompt method according to the first aspect.

In a fifth aspect, an embodiment of the present disclosure further provides a computer program, including computer-readable code, when the computer-readable code is executed in a computer device, a processor in the computer device executes the navigation for implementing the navigation described in the first aspect hint method.

In a sixth aspect, an embodiment of the present disclosure further provides a computer program product, including computer program instructions, the computer program instructions causing a computer to execute the navigation prompt method described in the first aspect.

In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings required in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present disclosure, and together with the description serve to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. Other related figures are obtained from these figures.

FIG. 1A shows a schematic diagram 1 of an application scenario of an electronic device provided by an embodiment of the present disclosure;

FIG. 1B shows a schematic diagram 2 of an application scenario of an electronic device provided by an embodiment of the present disclosure;

FIG. 2 shows a flowchart of a navigation prompt method provided by an embodiment of the present disclosure;

3 shows a flowchart of a method for judging whether there is a risk of collision between an AR device and a target object provided by an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of a scenario of an early warning prompt provided by an embodiment of the present disclosure;

5 shows a flowchart of another method for judging whether there is a risk of collision between an AR device and a target object provided by an embodiment of the present disclosure;

FIG. 6 shows a schematic diagram of another early warning prompt provided by an embodiment of the present disclosure;

7 shows a flowchart of a method for determining relative pose information between a target object and an AR device provided by an embodiment of the present disclosure;

FIG. 8 shows a schematic structural diagram of a navigation prompting device provided by an embodiment of the present disclosure;

FIG. 9 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. The components of the disclosed embodiments generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure as claimed, but is merely representative of selected embodiments of the disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present disclosure.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The term "and/or" in this paper only describes an association relationship, which means that there can be three kinds of relationships, for example, A and/or B, which can mean: the existence of A alone, the existence of A and B at the same time, the existence of B alone. a situation. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

In a general navigation scenario, a virtual indication arrow pointing to the target can be displayed in the navigation device, and the user can walk forward according to the virtual indication arrow, that is, the destination can be reached. In the case of an area, if the user still follows the arrows in the mechanical walking process, danger is likely to occur. Therefore, how to ensure the safety of the user during the navigation process is a technical problem to be solved by the present disclosure.

Based on the above research, the present disclosure provides a navigation prompt method, and proposes that when the AR device is located in the target area, target detection can be performed on the real scene image collected by the AR device. When there is a risk of collision between the AR device and the target object, the AR device can be used to warn the user to avoid safety accidents caused by the collision between the user and the target object during the navigation process, thereby improving the user's travel safety.

The navigation prompting method provided by the embodiment of the present disclosure can be executed by an electronic device.

In some possible implementations, as shown in FIG. 1A , the electronic device 10 may include a processor 11 and an image acquisition part 12 , in this way, the electronic device 10 can acquire an image of a real scene through the image acquisition part 12 , and use the processor 11 to acquire an image of a real scene. The real scene image is analyzed and processed to determine the position of the electronic device 10, and when it is detected that the electronic device 10 may collide with other objects, the user is alerted. For example, electronic devices may be implemented as AR devices, which may include devices capable of augmented reality, such as smartphones, tablets, and AR glasses.

In some possible implementations, as shown in FIG. 1B , the electronic device 10 may receive images of real scenes sent by other devices (such as AR devices) 30 through the network 20 . In this way, the electronic device 10 can locate the position of the AR device 30 based on the real scene image, and when detecting that the AR device 30 may collide with other objects, output warning information to the user through the human-computer interaction interface of the AR device 30 . For example, the electronic device may be implemented as a server.

Referring to FIG. 2, which is a flowchart of a navigation prompt method provided by an embodiment of the present disclosure, the navigation prompt method includes the following S101-S105:

S101, obtain a real scene image captured by an augmented reality AR device.

Exemplarily, the AR device may include devices capable of augmented reality, such as smartphones, tablet computers, and AR glasses. The AR device may have a built-in image acquisition component or an external image acquisition component. After the AR device enters the working state, it can collect images through the The component captures real-time images of the scene in real time.

S102, based on the real scene image, determine whether the AR device is located within the geographic location range corresponding to the target area.

Exemplarily, the target area may be a predetermined area with a certain driving danger, such as a road area, a landslide mountain road, a construction area, or other areas prone to danger.

Exemplarily, based on the real scene image, the AR device can be positioned to determine the first pose information of the AR device in the world coordinate system, and then based on the first pose information of the AR device, it is determined whether the AR device is located in the AR device. Within the geographic range corresponding to the target area.

In some embodiments of the present disclosure, the first pose information of the AR device may be determined based on the determined first pose information of the AR device and the geographic location range of each area included in the three-dimensional scene map representing the real scene in the world coordinate system, to determine the first position of the AR device. Whether the pose information is located in the geographic location range corresponding to the target area, if the first pose information of the AR device is located within the geographic location range corresponding to the target area, it can be determined that the AR device is located within the geographic location range corresponding to the target area, otherwise the AR device is located within the geographic location range corresponding to the target area. The device is not located within the geographic range of the target area.

Considering that the real scene image is collected by the image acquisition part of the AR device, the first pose information of the AR device can be represented by the first pose information of the image acquisition part of the AR device, which can include the image acquisition part of the AR device in the real world. The position coordinates and attitude data in the world coordinate system corresponding to the scene, where the position coordinates can be represented by the position coordinates of the image acquisition component in the world coordinate system; the attitude data can be represented by the orientation of the image acquisition component. The orientation can be represented by the included angles between the optical axis of the image acquisition component and the X, Y and Z axes in the world coordinate system.

In some embodiments of the present disclosure, when positioning the AR device based on the real scene image captured by the AR device, the AR device may be positioned based on the real scene image and the three-dimensional scene map representing the real scene. In addition, during the positioning process The AR device can also be positioned in combination with the built-in inertial measurement unit (IMU) of the AR device. The detailed positioning method will be introduced later.

S103, when the AR device is located within the geographic location range corresponding to the target area, detect whether there is a target object in the real scene image.

Exemplarily, the target object may be an obstacle that the user may encounter during driving, and may include static obstacles and dynamic obstacles. For example, if the target area is a road, the target object may include vehicles, trees, etc. , railings, etc.

Exemplarily, a pre-trained neural network for target detection can be used to detect whether a target object exists in an image of a real scene, and to determine the detection information of the target object, such as determining the corresponding position of the bounding rectangle of the target object in the image coordinate system. information, and the corresponding position information of the center point of the enclosing rectangular frame in the image coordinate system, or the pixel coordinate value of the pixel points constituting the target object in the image coordinate system, etc.

S104 , in the case where the target object exists in the real scene image, determine whether there is a risk of collision between the AR device and the target object.

S105 , in the case that there is a risk of collision between the AR device and the target object, a warning is given to the user through the AR device.

In the case where it is determined that the target object is detected from the real scene image, it can be further judged whether there is a risk of collision between the AR device and the target object. In the case of the risk of collision, the AR device is used to warn the user.

Exemplarily, when the AR is used to alert the user to the device, it may include:

The text prompt information, animation prompt information, and warning signs may be virtual objects superimposed on the real scene, and may be virtual objects superimposed on the real scene image captured by the AR device for prompting, so as to achieve the purpose of prompting the user.

In addition to prompting users with visual information such as text prompts, animation prompts, and warning signs, it can also prompt users in the form of voice and/or flashing, so as to effectively remind users to pay attention to the surrounding environment and improve travel safety for users .

The above S101 to S104 will be described in detail below with reference to the embodiments.

For the above S102, when judging whether the AR device is located within the geographic location range corresponding to the target area based on the real scene image, the following steps S1011 to S1013 may be included:

S1011, based on the real scene image and the preset three-dimensional scene map, determine the first pose information of the AR device in the world coordinate system; wherein, the three-dimensional scene map is a map representing the real scene to which the target area belongs;

S1012, based on the first pose information of the AR device and the geographic location range of each area associated with the three-dimensional scene map in the world coordinate system, determine whether the AR device is located within the geographic location range corresponding to the target area.

S1013, if the first pose information of the AR device indicates that the AR device is located within the geographic location range corresponding to the target area, determine that the AR device is located within the geographic location range corresponding to the target area, and if the first pose information of the AR device indicates that the AR device is located If it is not located within the geographic location range corresponding to the target area, it is determined that the AR device is not located within the geographic location range corresponding to the target area.

Exemplarily, a three-dimensional scene map representing the real scene can be generated by pre-shooting the video or image data obtained from the real scene. For details of the generation method, please refer to the following section. The three-dimensional scene map is generated based on the video data corresponding to the real scene, and can be constructed and A 3D scene map in which the scenes are completely overlapped in the same coordinate system, for example, a 3D scene map that completely overlaps with the real scene in the world coordinate system can be constructed. Therefore, the 3D scene map can be used as a high-precision map of the real scene.

Exemplarily, after obtaining the real scene image captured by the AR device, the first pose information of the AR device can be determined based on the real scene image and the pre-built three-dimensional scene map representing the real scene image. The detailed process will be described in It will be explained in detail later.

In addition, considering that real scene images are not collected in real time, they are generally collected at set time intervals. In addition, positioning based on real scene images and 3D scene maps consumes a lot of power. Therefore, when positioning AR devices , in the process of determining the first pose information of the AR device, the visual positioning based on the real scene image and the IMU positioning method can be used in combination.

Exemplarily, the first pose information of the AR device can be periodically determined according to the visual positioning, and the positioning is performed by the IMU in the intermediate process. For example, the positioning is performed visually every 10 seconds. The pose information, the first pose information at the 10th, 20th, and 30th seconds are obtained based on visual positioning, and the first pose information for the first second after starting work can be based on the first The pose information and the data collected by the IMU of the AR device from the start of work to the first second are estimated. Similarly, the first pose information in the second second can be based on the first pose information in the first second and It is estimated from the data collected by the IMU of the AR device from the first second to the second second. With the accumulation of time, when the first pose information obtained based on the IMU positioning method is no longer accurate, the visual positioning method can be used. Correction to obtain the first pose information with higher accuracy.

In addition, the positioning process of the AR device can also be based on the method of Simultaneous Localization And Mapping (SLAM) to locate the AR device, such as constructing a world coordinate system for the real scene in advance. Pre-acquire the first pose information of the AR device in the world coordinate system, and obtain the real scene image captured by the AR device, and construct a 3D scene map of the real scene in real time with the real scene image captured by the AR device during the movement process And locate the AR device to obtain the first pose information of the AR device at different times.

Exemplarily, each area occupies a certain area in the three-dimensional scene map, the boundary line of the area can be marked in advance, and the position coordinates of the position points on the boundary line in the world coordinate system can be obtained. The geographic extent of the region in the world coordinate system.

Exemplarily, the position coordinates of each position point on the boundary line under the world coordinate system may include the coordinate value of the position point along the X-axis direction under the world coordinate system, and the position point along the Y-axis direction under the world coordinate system. The coordinate value of the position point along the Z axis in the world coordinate system. In some embodiments of the present disclosure, when determining the geographic location range of the region in the world coordinate system according to the position coordinates of the location points on the boundary line in the world coordinate system, the location points on the boundary line in the world coordinate system may be used to determine the geographic location range of the region. The coordinate value along the X-axis direction below, determine the coordinate range of the area along the X-axis direction, and based on the coordinate value of the position point on the boundary line along the Y-axis direction in the world coordinate system, determine the area along the Y-axis direction. The coordinate range, and the coordinate value along the Z-axis direction of the position point on the boundary line in the world coordinate system, determine the coordinate range of the area along the Z-axis direction, the coordinate range of the area along the X-axis direction, along the Y-axis The coordinate range of the direction and the coordinate range along the Z-axis direction are used as the geographic location range corresponding to the area.

After obtaining the first pose information of the AR device, it can be determined whether the AR device is located in the geographic location corresponding to the target area according to the location coordinates of the AR device in the world coordinate system corresponding to the real scene and the geographic location range corresponding to each area. within the range.

In the embodiment of the present disclosure, it is proposed that the first pose information of the AR device can be determined according to the real scene image and the three-dimensional scene map representing the real scene image. According to the first pose information of the AR device and the geographic location range corresponding to each area, it can quickly determine whether the AR device is located in the target area to be alerted, such as on the road where vehicles pass, so as to ensure the user's safe travel.

In one embodiment, for the above S104, when there is a target object in the real scene image, when judging whether there is a risk of collision between the AR device and the target object, as shown in FIG. 3, the following steps S201-S202 are included:

S201 , based on a real scene image, determine relative pose information between a target object and an AR device.

Exemplarily, the relative pose information between the AR device and the target object may include the relative distance and relative angle between the AR device and the target object, wherein the relative distance The relative distance of the target position point of the target object in the world coordinate system, the target position point may include the center position point of the target object, or the closest distance between the boundary of the target object and the optical center of the image acquisition component The relative angle can be represented by the angle between the direction when the optical axis of the image acquisition component of the AR device points to the target position point of the target object and the orientation of the optical axis.

S202, based on the relative pose information, determine whether there is a risk of collision between the AR device and the target object.

Exemplarily, whether there is a risk of collision between the AR device and the target object may be determined based on the relative distance and relative angle, and the manner of determining whether there is a risk of collision will be described later.

Exemplarily, the relative pose information may include the pose of the target object relative to the AR device, and the pose of the AR device relative to the target object, such as the target object approaching the AR device, or the AR device approaching the target object, or both. They are both approaching each other, and it can be determined that the relative poses of the two are getting closer and closer. These can be determined through the real scene images collected by the AR device. The determination process is described later.

In one embodiment, the target object includes a static obstacle. For the above S202, when judging whether there is a risk of collision between the AR device and the target object based on the relative pose information, the following steps S2021 to S2023 are included:

S2021, based on the relative pose information, determine whether the distance between the AR device and the static obstacle is less than a first preset distance.

Exemplarily, the static obstacles may be static obstacles, such as railings, trees, steps, etc., collected by the AR device while the user is driving in the target area.

Exemplarily, the first preset distance may be set according to experience, so that the user can be prompted through the AR device in advance to avoid collision with a static obstacle when the user continues to drive in the current driving direction.

S2022, in the case that the distance between the AR device and the static obstacle is less than the first preset distance, determine whether the AR device faces the static obstacle.

S2023, when the AR device faces the static obstacle, determine that there is a risk of collision between the AR device and the static obstacle.

Exemplarily, the position coordinates of the target object in the camera coordinate system corresponding to the AR device can be determined according to the real scene image collected by the AR device, and then according to the direction of the optical axis of the AR device, it can be determined whether the AR device is facing a static obstacle. thing.

Exemplarily, in the case where there is a risk of collision between the AR device and the static obstacle, the risk level can be determined based on the distance between the AR device and the static obstacle. When the AR device prompts the user, it can give corresponding early warning prompts according to different risk levels. For example, when the risk level is low, the AR device can provide early warning prompts in one way. When the risk level is high, you can use the AR The equipment provides early warning prompts in various ways.

Exemplarily, as shown in Figure 4, it is a schematic diagram of a scene where the target area is a construction area and the target object is a railing, and the user is prompted by an AR device. Display virtual text information used to warn the user, such as "There is a railing ahead, please pay attention to avoid collision", to remind the user of danger ahead, please pay attention to safety, so as to remind the user to stop approaching or take a detour.

In another embodiment, the target object includes dynamic obstacles. For the above S202, when judging whether there is a risk of collision between the AR device and the target object based on the relative pose information, as shown in FIG. 5 , it may include The following S301～S302:

S301 , based on a plurality of real scene images, determine relative motion information between the AR device and the dynamic obstacle.

Exemplarily, the relative movement information may include the movement direction of the AR device relative to the dynamic obstacle, or the relative movement direction of the dynamic obstacle relative to the AR device, and may also include the relative movement speed.

Exemplarily, based on a single real scene image, the relative pose information between the AR device and the dynamic obstacle can be determined, and the AR device and the dynamic obstacle can be determined through the relative pose information corresponding to the multiple real scene images respectively. For example, based on two adjacent real scene images, the position coordinates of the dynamic obstacle in the camera coordinate system corresponding to the AR device can be determined, and then the driving direction and the driving direction of the dynamic obstacle relative to the AR device can be determined. Driving speed.

S302, based on the relative pose information and the relative motion information, determine whether there is a risk of collision between the AR device and the dynamic obstacle.

Exemplarily, in the case where the target object is a dynamic obstacle, considering that the dynamic obstacle can move in the target area, the AR device can also move in the target area, when the two are close to each other, It can be determined that there is a risk of collision between the AR device and the dynamic obstacle, and conversely, it can be determined that there is no risk of collision between the AR device and the dynamic obstacle.

For the above S302, when judging whether there is a risk of collision between the AR device and the dynamic obstacle based on the relative pose information and the relative motion information, the following steps S3021 to S3022 are included:

S3021 , based on the relative pose information, determine whether the distance between the AR device and the dynamic obstacle is less than a second preset distance.

Exemplarily, the dynamic obstacles may include vehicles, pedestrians, etc., and the second preset distance may be set based on experience, subject to a safety distance corresponding to sufficient obstacle avoidance time for the user, such as the target object needs to avoid obstacles. The duration is n1 seconds, and the target object can travel m meters within n1 seconds. In this case, the setting of the second preset distance needs to be greater than m meters.

Exemplarily, considering that when the target object is a dynamic obstacle, if the AR device and the target object move towards each other, the risk of collision is greater than that when the target object is a static obstacle. Therefore, the second The preset distance needs to be larger than the above-mentioned first preset distance during setting.

S3022, in the case that the distance between the AR device and the dynamic obstacle is less than the second preset distance, based on the relative motion information, determine the distance between the driving direction of the dynamic obstacle relative to the AR device and the direction of the dynamic obstacle toward the AR device Whether the included angle is less than the set angle threshold.

Exemplarily, based on the real scene image collected by the AR device at the last moment, the position coordinates of the dynamic obstacle in the camera coordinate system corresponding to the AR device at the last moment can be determined, and then based on the real scene image collected by the AR device at the current moment, Determine the position coordinates of the dynamic obstacle in the camera coordinate system corresponding to the AR device at the current moment. In this way, according to the position coordinates corresponding to the dynamic obstacle at the previous moment and the position coordinates corresponding to the current moment, the relative position of the dynamic obstacle to the AR device can be determined. For example, point the position coordinates corresponding to the dynamic obstacle at the last moment in the direction of the position coordinates corresponding to the dynamic obstacle at the current moment, as the driving direction of the dynamic obstacle relative to the AR device.

Exemplarily, the direction of the dynamic obstacle towards the AR device can be determined by the position coordinates of the dynamic obstacle in the camera coordinate system where the AR device is located, and the position coordinates of the AR device in the camera coordinate system, for example, the dynamic obstacle The position coordinates in the camera coordinate system where the AR device is located point to the direction of the position coordinates of the AR device in the camera coordinate system as the direction of the dynamic obstacle toward the AR device.

Exemplarily, the set angle threshold can be set empirically to determine whether the dynamic obstacle and the AR device are gradually approaching, for example, between the driving direction of the dynamic obstacle relative to the AR device and the direction of the dynamic obstacle toward the AR device. When the included angle is less than the set angle threshold, it can be determined that the dynamic obstacle and the AR device are gradually approaching.

S3023, when the included angle is smaller than the set angle threshold, determine that there is a risk of collision between the AR device and the dynamic obstacle.

When the dynamic obstacle and the AR device are gradually approaching, it can be estimated that there is a risk of collision between the AR device and the dynamic obstacle.

Exemplarily, in the case where there is a risk of collision between the AR device and the dynamic obstacle, the risk level may be determined based on the distance and the relative movement speed between the AR device and the dynamic obstacle. The higher the movement speed, the higher the risk level. For example, the mapping relationship between different distances and risk scores, as well as the mapping relationship between different relative movement speeds and risk scores can be established in advance. After obtaining the distance and relative movement speed between the AR device and the dynamic obstacle, The current risk score can be determined according to the predetermined mapping relationship between different distances and risk scores, and the mapping relationship between different relative motion speeds and risk scores, and further, the risk level can be determined according to the risk score.

In some embodiments of the present disclosure, when an AR device is used to provide an early warning to a user, the corresponding early warning can be prompted according to different risk levels. When the risk level is high, early warnings can be given in various ways through AR devices.

Exemplarily, as shown in Figure 6, it is a schematic diagram of a scene where the target area is a road and the target object is a vehicle, and the user is prompted by the AR device. When the user approaches the vehicle, the user can display it on the display screen of the AR device. Virtual text information used to warn the user, such as "there is a vehicle in the road ahead, please pay attention to avoid collision", to remind the user of danger ahead, please pay attention to safety, so as to remind the user to stop approaching or take a detour.

Regarding the relative pose information between the target object and the AR device mentioned above, in one embodiment, when determining the relative pose information between the target object and the AR device based on the real scene image, as shown in FIG. 7 display, may include the following S401-S403:

S401, according to the real scene image, the detection area corresponding to the target object in the real scene image, and the preset three-dimensional scene map, determine the second pose information of the target object in the world coordinate system; wherein, the three-dimensional scene map represents the target A map of the real scene to which the area belongs;

Exemplarily, the three-dimensional scene map here and the above-mentioned three-dimensional scene map may be the same scene map, and the construction process is described in the following description.

S402, the AR device is positioned to obtain the first pose information of the AR device in the world coordinate system.

Exemplarily, the process of locating the AR device here may be similar to the process of determining the first pose information of the AR device in S1011, as described later, or it may be determined according to the positioning sensor set by the AR device itself, such as the one set by itself. Global satellite navigation equipment and inertial measurement units.

S403, based on the first pose information and the second pose information, determine relative pose information between the target object and the AR device.

Exemplarily, the three-dimensional scene map representing the real scene includes multiple feature points in the real scene, and the multiple feature points of the real scene can also be extracted from the real scene image, so that based on the multiple feature points included in the real scene image And the multiple feature points contained in the 3D scene map are matched to the feature points, and the position coordinates of the multiple feature points that constitute the target object in the corresponding world coordinate system in the 3D scene map are determined, that is, the target object contained in the detection area is obtained. The position coordinates of the feature points in the world coordinate system, based on which, the second pose information of the target object in the world coordinate system can be determined.

After obtaining the second pose information of the target object in the world coordinate system, it can be determined based on the first pose information of the AR device in the world coordinate system and the second pose information of the target object in the world coordinate system The relative pose information between the target object and the AR device is obtained.

In one embodiment, when determining the relative pose information between the target object and the AR device based on the real scene image, the method may further include:

According to the first pose information of the AR device in the world coordinate system obtained after positioning the AR device, and the pose information of the target object in the image coordinate system corresponding to the real scene image, determine the detected target object and the AR device. relative pose information.

Exemplarily, based on the pose information of the target object in the image coordinate system corresponding to the real scene image, and the corresponding internal parameters of the image acquisition component of the AR device, the pose of the target object in the camera coordinate system corresponding to the AR device can be determined. information. Further, based on the first pose information of the AR device in the world coordinate system obtained after positioning the AR device, the external parameters corresponding to the image acquisition component of the AR device can be determined, and based on the external parameters corresponding to the image acquisition component of the AR device, it can be determined. The second pose information of the target object in the world coordinate system. Finally, based on the second pose information of the AR device and the target object in the world coordinate system, the relative pose information of the AR device and the target object in the world coordinate system can be determined.

In some embodiments of the present disclosure, according to the first pose information of the AR device in the world coordinate system obtained after positioning the AR device, and the pose information of the target object in the image coordinate system corresponding to the real scene image, determine When detecting the relative pose information between the target object and the AR device, the following processes S501 to S505 are included:

S501 , perform target detection on the real scene image, and determine the pose information of each pixel that constitutes the target object in the real scene image in the image coordinate system.

Here, a pre-trained target detection neural network can be used to perform target detection on real scene images, determine the target object contained in the real scene image, and the pose information of each pixel that constitutes the target object in the image coordinate system, where The pose information of the pixel point in the image coordinate system refers to the pixel coordinate value of the pixel point in the image coordinate system.

S502 , based on the real scene image, determine a depth image corresponding to the real scene image; the depth image includes depth information of each pixel that constitutes the target object in the real scene image.

Exemplarily, the depth image corresponding to the real scene image can be determined according to the collected real scene image and the pre-trained neural network for determining the depth image, so as to obtain the depth image of each pixel that constitutes the target object in the real scene image. in-depth information.

S503, based on the pose information of each pixel that constitutes the target object in the real scene image in the image coordinate system, the depth information of the pixel, and the parameter information of the image acquisition component in the AR device, determine that the pixel is in world coordinates 3D coordinate information under the system.

Exemplarily, an image coordinate system can be established for the collected real scene image, and the pose information of each pixel that constitutes the target object in the image coordinate system can be determined based on the constructed image coordinate system, that is, the pose information in the image coordinate system can be determined. Pixel coordinate value.

The parameter information of the image acquisition part may include internal parameters and external parameters of the image acquisition part of the AR device, wherein the internal parameters are fixed parameters of the image acquisition part of the AR device, and the external parameters can be determined by the first pose information of the AR device, Among them, the internal parameter can be used to convert the coordinate value of the pixel in the image coordinate system into the coordinate value in the camera coordinate system; the external parameter can be used to convert the coordinate value of the pixel in the camera coordinate system into the coordinate value in the world Coordinate values in the coordinate system.

Exemplarily, through the internal parameters of the image acquisition component in the AR device, the pixel coordinate value of each pixel that constitutes the target object in the image coordinate system can be converted to the X-axis and Y-axis directions in the camera coordinate system. Based on the depth information of the pixel point, the coordinate value of the pixel point along the Z-axis direction in the camera coordinate system can be obtained, and then based on the external parameters of the image acquisition component, the coordinate value of the pixel point in the camera coordinate system can be obtained. Convert to the coordinate value in the world coordinate system, that is, get the three-dimensional coordinate information of the pixel point in the world coordinate system.

S504, based on the three-dimensional coordinate information of each pixel constituting the target object in the world coordinate system, determine the second pose information of the target object in the world coordinate system.

Exemplarily, after obtaining the three-dimensional coordinate information of each pixel constituting the target object in the world coordinate system, the three-dimensional detection information corresponding to the target object may be determined based on a pre-trained three-dimensional target detection neural network.

For example, based on the three-dimensional coordinate information of each pixel that constitutes the target object in the world coordinate system, the corresponding three-dimensional point cloud data of the target object can be obtained, and then based on the three-dimensional point cloud data and the pre-built three-dimensional target detection neural network, to The three-dimensional detection information corresponding to the target object is determined, and then the second pose information of the target object in the world coordinate system is determined according to the three-dimensional detection information.

Exemplarily, the three-dimensional detection information of the target object may include the position coordinates of the center point of the target object in the world coordinate system, the length, width and height of the 3D detection frame of the target object, and the set positive direction and world coordinates of the target object. The angle between each coordinate axis of the system, the set positive direction of the target object can be the direction of the head of the car, and the position coordinates of the center point of the target object in the world coordinate system and the direction of the head of the target object can be used as the The second pose information of the target object in the world coordinate system.

S505, according to the first pose information of the AR device under the world coordinate system obtained after positioning the AR device, and the determined second pose information of the target object under the world coordinate system, determine the detected target object and the AR Relative pose information between devices.

In another embodiment, real-time mapping and positioning can also be performed based on SLAM to obtain a three-dimensional scene map of the real scene where the AR device is located and the first pose information of the AR device. The three-dimensional scene map can obtain the second pose information of the target object in the world coordinate system, so that the relationship between the AR device and the target object can be determined according to the first pose information of the AR device and the second pose information of the target object. Relative pose information.

The three-dimensional scene map mentioned above can be pre-built in the following ways, including S601 to S603:

S601, acquiring a plurality of real scene sample images.

Exemplarily, a multi-angle aerial photography of the real scene, such as a certain city, can be performed in advance by a drone, and a large number of real scene sample images corresponding to the real scene can be obtained.

S602 , constructing an initial three-dimensional scene model representing the real scene based on the plurality of real scene sample images.

For S602, when generating an initial three-dimensional scene model corresponding to a real scene based on a plurality of real scene sample images, the following steps S6021 to S6022 may be included:

S6021, extracting multiple feature points from each obtained sample image of the real scene;

S6022, an initial 3D scene model is generated based on the extracted multiple feature points and the pre-stored 3-D sample map matching the real scene; wherein, the 3-D sample map is a pre-stored 3-D map representing the appearance features of the real scene.

In some embodiments of the present disclosure, the feature points extracted for each real scene sample image may be points that can represent key information of the real scene sample image. For example, for a real scene sample image including buildings, the feature points here may represent The feature points of the building outline information.

Exemplarily, the pre-stored 3D sample map of the real scene may include a pre-set 3D map that can characterize the morphological features of the real scene and is dimensioned, such as a computer that characterizes the morphological features of the real scene. Aided design (Computer Aided Design, CAD) three-dimensional drawing.

For the real scene, when enough feature points are extracted, the feature point cloud composed of the feature points can form a three-dimensional model representing the real scene. The feature points in the feature point cloud here are unitless, and the feature point cloud is composed of The 3D model is also unitless. Then, after aligning the feature point cloud with a 3D map with scale annotations that can characterize the topographical features of the real scene, the initial 3D scene model corresponding to the real scene is obtained.

S603: Align the calibration feature points on the constructed initial three-dimensional scene model with the calibration feature points corresponding to the real scene to generate a three-dimensional scene map.

The generated initial 3D model may be distorted, and then the alignment process can be completed through the calibration feature points on the real scene and the initial 3D scene model, so that a 3D scene model with high accuracy can be obtained.

For S603, when aligning the calibration feature points on the constructed initial 3D scene model with the calibration feature points corresponding to the real scene, and generating a 3D scene map, include S6031-S6032:

S6031, extracting, in the initial three-dimensional scene model corresponding to the real scene, calibration feature points used to represent multiple spatial position points of the real scene;

S6032: Determine the real position coordinates of the calibration feature points in the real two-dimensional map corresponding to the real scene, and adjust the position coordinates of each feature point in the initial three-dimensional scene model based on the real position coordinates corresponding to each calibration feature point.

Exemplarily, some feature points representing the spatial position points of the edges and corners of the building can be selected as the calibration feature points here, and then based on the real position coordinates corresponding to the calibration feature points and the position of the calibration feature point in the initial three-dimensional scene model. Coordinates, determine the position coordinate adjustment amount, and then correct the position coordinates of each feature point in the initial three-dimensional model based on the position coordinate adjustment amount, so that a three-dimensional scene model with high accuracy can be obtained.

Exemplarily, the real position coordinates corresponding to the calibration feature points here may refer to the position coordinates of the world coordinate system corresponding to the real scene to which the specified feature points belong in the target area. After alignment, the obtained three-dimensional scene map contains the world coordinates of each feature point. Position coordinates under the system.

After the 3D scene map of the real scene is constructed, the AR device can be positioned based on the real scene image captured by the AR device and the 3D scene map, and the first pose information of the AR device can be determined based on the real scene image and the 3D scene map. can include the following S701-S703:

S701, extract the feature points contained in the real scene image, and extract the feature points of each real scene sample image when the three-dimensional scene map is pre-built;

S702, based on the feature points corresponding to the real scene image and the feature points corresponding to each real scene sample image when the three-dimensional scene map is pre-built, determine the target real scene sample image with the highest similarity with the real scene image;

S703: Determine the first pose information of the AR device based on the shooting pose information corresponding to the target real scene sample image.

Exemplarily, after the real scene image captured by the AR device is acquired, the feature points in the real scene image and the feature points of each real scene sample image when the three-dimensional scene map is pre-built can be used to find the image similar to the real scene image. The sample image of the target real scene with the highest degree. For example, based on the feature information of the feature points of the real scene image and the feature information of the feature points of each real scene sample image, the similarity value of the real scene image and each real scene sample image can be determined, and the similarity value is the highest and exceeds The real scene sample image with the similarity threshold is used as the target real scene sample image here.

After the target real scene sample image is determined, the first pose information of the AR device may be determined based on the shooting pose information corresponding to the target real scene sample image.

In some embodiments of the present disclosure, for the above S703, when determining the first pose information of the AR device based on the shooting pose information corresponding to the target real scene sample image, the following steps S7031 to S7032 may be included:

S7031, determine the relative pose information between the target object in the target real scene sample image and the target object in the real scene image;

S7032: Determine the first pose information of the AR device based on the relative pose information and the shooting pose information corresponding to the target real scene sample image.

Exemplarily, the target object contained in the target real scene sample image with the highest similarity with the real scene image is the same target object as the target object contained in the real scene image, for example, the target object contained in the real scene image is a building. A. The target object contained in the target real scene sample image is also building A. In this way, by determining the relative pose information between the building A in the real scene image and the building A in the target real scene sample image, determine Obtain the relative shooting pose information of the image acquisition component when shooting the real scene image and the shooting target real scene sample image, and further determine the AR device based on the relative shooting pose information and the shooting pose information corresponding to the target real scene sample image. first pose information.

Exemplarily, when determining the relative pose data between the target object in the target real scene sample image and the target object in the real scene image, the target real scene sample image and the target real scene sample image and The three-dimensional detection information corresponding to the target object in the real scene image is determined, and then the relative pose data is determined through the respectively determined target real scene sample image and the three-dimensional detection information corresponding to the target object in the real scene image.

In special cases, when the pose information of the target object in the target real scene sample image is the same as that of the target object in the real scene image, the shooting pose information corresponding to the target real scene sample image can be directly used as the first AR device here. One pose information.

Those skilled in the art can understand that in the above method of the embodiments, the writing order of each step does not mean a strict execution order but constitutes any limitation to the implementation process, and the execution order of each step should be based on its function and possible internal logic. Sure.

Based on the same technical concept, the embodiments of the present disclosure also provide a navigation prompting device corresponding to the navigation prompting method. Reference may be made to the implementation of the method, and repeated descriptions will not be repeated.

Referring to FIG. 8 , which is a schematic diagram of a navigation prompt device 800 provided by an embodiment of the present disclosure, the navigation prompt device includes:

The image acquisition part 801 is configured to acquire the real scene image captured by the augmented reality AR device;

The first judging part 802 is configured to judge whether the AR device is located within the geographic location range corresponding to the target area based on the real scene image;

The target detection part 803 is configured to detect whether there is a target object in the real scene image when the AR device is located within the geographic location range corresponding to the target area;

The second judging part 804 is configured to judge whether there is a risk of collision between the AR device and the target object when the target object exists in the real scene image;

The early warning prompting part 805 is configured to provide early warning prompting to the user through the AR device when there is a risk of collision between the AR device and the target object.

In a possible implementation, the first judging part 802 is further configured to determine the first pose information of the AR device in the world coordinate system based on the real scene image and the preset three-dimensional scene map of the table; wherein the three-dimensional The scene map is a map that represents the real scene to which the target area belongs; based on the first pose information of the AR device and the geographic location of each area associated with the 3D scene map in the world coordinate system, it is determined whether the AR device is located in the corresponding target area. within the geographic range.

In a possible implementation, the second judgment part 804 is further configured to determine the relative pose information between the target object and the AR device based on the real scene image; based on the relative pose information, determine the AR device and the target object Whether there is a risk of collision between them.

In a possible implementation, the target object includes a static obstacle, and the second judgment part 804 is further configured to judge whether the distance between the AR device and the static obstacle is less than the first preset distance based on the relative pose information ; When the distance between the AR device and the static obstacle is less than the first preset distance, determine whether the AR device is facing the static obstacle; if the AR device is facing the static obstacle, determine whether the AR device is facing the static obstacle. There is a risk of collision between them.

In a possible implementation, the target object includes a dynamic obstacle, and the second judging part 804 is further configured to determine relative motion information between the AR device and the dynamic obstacle based on a plurality of real scene images; Attitude information and relative motion information to determine whether there is a risk of collision between the AR device and dynamic obstacles.

In a possible implementation, the second judging part 804 is further configured to judge whether the distance between the AR device and the dynamic obstacle is less than the second preset distance based on the relative pose information; When the distance between the objects is less than the second preset distance, based on the relative motion information, determine whether the angle between the driving direction of the dynamic obstacle relative to the AR device and the direction of the dynamic obstacle toward the AR device is less than the set angle Threshold; when the included angle is less than the set angle threshold, it is determined that there is a risk of collision between the AR device and the dynamic obstacle.

In a possible implementation, the second judging part 804 is further configured to determine the world coordinates of the target object according to the real scene image, the detection area corresponding to the target object in the real scene image, and the preset three-dimensional scene map The second pose information under the system; among them, the three-dimensional scene map is a map representing the real scene to which the target area belongs; locate the AR device to obtain the first pose information of the AR device in the world coordinate system; based on the first The pose information and the second pose information determine the relative pose information between the target object and the AR device.

In a possible implementation, the early warning prompting part 804 is further configured to provide early warning prompts to the user in at least one of voice form, text form, animation form, warning symbol, and flashing form through the AR device.

For the description of the processing flow of each part in the apparatus and the interaction flow between the various parts, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

Corresponding to the navigation prompting method in FIG. 2 , an embodiment of the present disclosure further provides an electronic device 900 . As shown in FIG. 9 , the schematic structural diagram of the electronic device 900 provided by the embodiment of the present disclosure includes:

The processor 91, the memory 92, and the bus 93; the memory 92 is used to store the execution instructions, including the memory 921 and the external memory 922; the memory 921 here is also called the internal memory, which is used to temporarily store the operation data in the processor 91, and For the data exchanged by the external memory 922 such as the hard disk, the processor 91 exchanges data with the external memory 922 through the memory 921. When the electronic device 900 is running, the communication between the processor 91 and the memory 92 is through the bus 93, so that the processor 91 executes the following instructions : Obtain the real scene image captured by the augmented reality AR device; based on the real scene image, determine whether the AR device is located within the geographic location range corresponding to the target area; if the AR device is located within the geographic location range corresponding to the target area, detect the real scene Whether there is a target object in the image; if there is a target object in the real scene image, determine whether there is a risk of collision between the AR device and the target object; if there is a risk of collision between the AR device and the target object, through the AR device Alert users.

Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the navigation prompting method described in the above method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

Embodiments of the present disclosure further provide a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the steps configured to implement the above-mentioned navigation prompting method.

An embodiment of the present disclosure further provides a computer program product, where the computer program product includes computer program instructions, and the computer program instructions can be used to execute the steps of the navigation prompt method described in the above method embodiments. For details, please refer to the above method embodiments , and will not be repeated here.

Wherein, the above-mentioned computer program product can be realized by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments, and details are not repeated here. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the parts that contribute to the prior art or the parts of the technical solutions. The computer software products are stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

Finally, it should be noted that the above-mentioned embodiments are only embodiments of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, but not to limit them. The present disclosure is described in detail by the examples, and those of ordinary skill in the art should understand that: any person skilled in the art can still modify or modify the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure. Changes can be easily conceived, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered by the present disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

A navigation prompt method comprising:

Obtain real scene images captured by augmented reality AR devices;

Based on the real scene image, determine whether the AR device is located within the geographic location range corresponding to the target area;

In the case that the AR device is located within the geographic location range corresponding to the target area, detecting whether there is a target object in the real scene image;

In the case that there is a target object in the real scene image, determine whether there is a risk of collision between the AR device and the target object;

In the case that there is a risk of collision between the AR device and the target object, an early warning prompt is given to the user through the AR device.
The navigation prompting method according to claim 1, wherein the determining whether the AR device is located within a geographic location range corresponding to a target area based on the real scene image comprises:

Based on the real scene image and the preset three-dimensional scene map, determine the first pose information of the AR device in the world coordinate system; the three-dimensional scene map is a map representing the real scene to which the target area belongs;

Based on the first pose information and the geographic location range of each area associated with the three-dimensional scene map in the world coordinate system, it is determined whether the AR device is located within the geographic location range corresponding to the target area.
The navigation prompting method according to claim 1 or 2, wherein the determining whether there is a risk of collision between the AR device and the target object when there is a target object in the real scene image comprises:

Based on the real scene image, determine the relative pose information between the target object and the AR device;

Based on the relative pose information, it is determined whether there is a risk of collision between the AR device and the target object.
The navigation prompting method according to claim 3, wherein the target object includes a static obstacle, and based on the relative pose information, it is determined whether there is a risk of collision between the AR device and the target object ,include:

Based on the relative pose information, determine whether the distance between the AR device and the static obstacle is less than a first preset distance;

In the case where the distance between the AR device and the static obstacle is less than a first preset distance, determine whether the AR device is facing the static obstacle;

In the case where the AR device faces the static obstacle, it is determined that there is a risk of collision between the AR device and the static obstacle.
The navigation prompting method according to claim 3 or 4, wherein the target object includes a dynamic obstacle, and it is determined whether there is a collision between the AR device and the target object based on the relative pose information risks, including:

Determine relative motion information between the AR device and the dynamic obstacle based on a plurality of real scene images;

Based on the relative pose information and the relative motion information, it is determined whether there is a risk of collision between the AR device and the dynamic obstacle.
The navigation prompting method according to claim 5, wherein the determining whether there is a risk of collision between the AR device and the dynamic obstacle based on the relative pose information and the relative motion information, comprising: :

Based on the relative pose information, determine whether the distance between the AR device and the dynamic obstacle is less than a second preset distance;

In the case that the distance between the AR device and the dynamic obstacle is less than the second preset distance, based on the relative motion information, it is determined that the driving direction of the dynamic obstacle relative to the AR device is different from that of the dynamic obstacle. Whether the included angle between the directions of the dynamic obstacles toward the AR device is less than the set angle threshold;

In the case that the included angle is smaller than the set angle threshold, it is determined that there is a risk of collision between the AR device and the dynamic obstacle.
The navigation prompting method according to any one of claims 3 to 6, wherein the determining relative pose information between the target object and the AR device based on the real scene image comprises:

According to the real scene image, the detection area corresponding to the target object in the real scene image, and the preset three-dimensional scene map, determine the second pose information of the target object in the world coordinate system; the three-dimensional The scene map is used to represent the map of the real scene to which the target area belongs;

Positioning the AR device to obtain the first pose information of the AR device in the world coordinate system;

Determine relative pose information between the target object and the AR device according to the first pose information and the second pose information.
The navigation prompting method according to any one of claims 1 to 7, wherein the performing an early warning prompt to the user through the AR device comprises:

An early warning prompt is given to the user through the AR device in at least one of a voice form, a text form, an animation form, a warning sign, and a flashing form.
A navigation prompt device, comprising:

The image acquisition part is configured to acquire the real scene image captured by the augmented reality AR device;

a first judging part, configured to judge whether the AR device is located within the geographic location range corresponding to the target area based on the real scene image;

a target detection part, configured to detect whether a target object exists in the real scene image when the AR device is located within the geographic location range corresponding to the target area;

The second judgment part is configured to judge whether there is a risk of collision between the AR device and the target object when there is a target object in the real scene image;

The early warning prompting part is configured to provide early warning prompting to the user through the AR device when there is a risk of collision between the AR device and the target object.
The apparatus according to claim 9, wherein the first determination part is further configured to determine the first position of the AR device in the world coordinate system based on the real scene image and the preset three-dimensional scene map posture information; the three-dimensional scene map is a map representing the real scene to which the target area belongs; based on the first posture information and the geographic location range of each area associated with the three-dimensional scene map in the world coordinate system, Determine whether the AR device is located within the geographic location range corresponding to the target area.
The apparatus according to claim 9 or 10, wherein the second judgment part is further configured to determine the relative pose information between the target object and the AR device based on the real scene image; The relative pose information determines whether there is a risk of collision between the AR device and the target object.
The apparatus of claim 11, wherein the target object comprises a static obstacle;

The second judging part is further configured to judge whether the distance between the AR device and the static obstacle is less than a first preset distance based on the relative pose information; If the distance between static obstacles is less than the first preset distance, determine whether the AR device is facing the static obstacle; if the AR device is facing the static obstacle, determine whether the AR device is facing the static obstacle There is a risk of collision with the static obstacle.
The apparatus of claim 11 or 12, wherein the target object comprises a dynamic obstacle;

The second judgment part is further configured to determine relative motion information between the AR device and the dynamic obstacle based on a plurality of real scene images; based on the relative pose information and the relative motion information, Determine whether there is a risk of collision between the AR device and the dynamic obstacle.
The apparatus according to claim 13, wherein the second determination part is further configured to determine whether the distance between the AR device and the dynamic obstacle is less than a second predetermined distance based on the relative pose information Set the distance; when the distance between the AR device and the dynamic obstacle is less than the second preset distance, determine the driving direction of the dynamic obstacle relative to the AR device based on the relative motion information Whether the included angle with the direction of the dynamic obstacle towards the AR device is less than a set angle threshold; if the included angle is less than the set angle threshold, determine the AR device and the dynamic obstacle There is a risk of collision between them.
The apparatus according to any one of claims 11 to 14, wherein the second judgment part is further configured to be based on the real scene image, the detection area corresponding to the target object in the real scene image, and A preset three-dimensional scene map, to determine the second pose information of the target object in the world coordinate system; the three-dimensional scene map is used to represent the map of the real scene to which the target area belongs; to locate the AR device , obtain the first pose information of the AR device in the world coordinate system; determine the relationship between the target object and the AR device according to the first pose information and the second pose information Relative pose information.
The device according to any one of claims 9 to 15, wherein the early warning prompt part is further configured to at least one of voice form, text form, animation form, warning sign, and flash form through the AR device Alert users.
An electronic device, comprising: a processor, a memory and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus , when the machine-readable instructions are executed by the processor, the navigation prompting method according to any one of claims 1 to 8 is executed.
A computer-readable storage medium storing a computer program on the computer-readable storage medium, the computer program executing the navigation prompt method according to any one of claims 1 to 8 when the computer program is run by a processor.
A computer program comprising computer readable code, when the computer readable code is run in an electronic device, a processor in the computer device executes the navigation prompt for implementing any one of claims 1 to 8 method.
A computer program product comprising computer program instructions that cause a computer to perform the navigation prompting method of any one of claims 1 to 8.