CN114299263A

CN114299263A - Display method and device for augmented reality AR scene

Info

Publication number: CN114299263A
Application number: CN202111673087.5A
Authority: CN
Inventors: 李斌; 欧华富; 李颖楠
Original assignee: Beijing Mianbaitang Intelligent Technology Co ltd
Current assignee: Beijing Mianbaitang Intelligent Technology Co ltd
Priority date: 2021-12-31
Filing date: 2021-12-31
Publication date: 2022-04-08
Also published as: WO2023124691A1

Abstract

The display method comprises the steps of responding to the recognition of a target identification object from a current scene image, determining second position and posture data of a virtual object in a screen coordinate system of AR equipment according to first position and posture data of the virtual object corresponding to the target identification object in a world coordinate system, displaying the virtual object on the AR equipment based on the second position and posture data, and responding to the fact that the target identification object is not recognized from the current scene image in the display process of the virtual object, and displaying the virtual object on a screen of the AR equipment based on preset target position and posture data in the screen coordinate system.

Description

Display method and device for augmented reality AR scene

Technical Field

The disclosure relates to the technical field of augmented reality, in particular to a display method and device of an augmented reality scene.

Background

Augmented Reality (AR) technology superimposes a virtual model on the real world, so that a real environment and a virtual object are presented on the same screen or space in real time. For the AR scene, how to enrich the interaction between the virtual object and the real scene, optimizing the display effect is one of the main research directions.

Disclosure of Invention

The embodiment of the disclosure provides a display method and device for an Augmented Reality (AR) scene, AR equipment and a storage medium.

In a first aspect, an embodiment of the present disclosure provides a method for displaying an augmented reality scene, where the method includes:

in response to the identification of a target identification object from a current scene image, determining second position and orientation data of a virtual object in a screen coordinate system of the AR equipment according to first position and orientation data of the virtual object corresponding to the target identification object in a world coordinate system;

presenting the virtual object at the AR device based on the second pose data;

in the process of displaying the virtual object, in response to the target identification object not being identified from the current scene image, displaying the virtual object on the screen of the AR device based on preset target pose data in the screen coordinate system.

In some embodiments, said presenting, during the presenting of the virtual object, the virtual object on the screen of the AR device based on preset target pose data in the screen coordinate system in response to the target identification object not being identified from the current scene image, includes:

in the display process of the virtual object, in response to the target identification object not being identified from the current scene image, determining the current displayed progress;

continuing to present the virtual object on the screen of the AR device based on the target pose data and the presented progress.

In some embodiments, the display method of the present disclosure further comprises:

in response to re-identifying the target identifying object from the current scene image, re-performing the step of presenting the virtual object at the AR device based on the second pose data.

displaying the virtual object with a first display effect in the case that the target identification object is identified from the current scene image;

in the event that the target identification object is not identified from the current scene image, presenting the virtual object with a second presentation effect, the second presentation effect being different from the first presentation effect.

In some embodiments, the target pose data comprises presentation position data and presentation pose data for the virtual object; the displaying the virtual object on the screen of the AR device based on preset target pose data in the screen coordinate system comprises:

at the screen position indicated by the presentation position data, presenting the virtual object in the pose indicated by the presentation pose data.

In some embodiments, the determining, according to the first pose data of the virtual object corresponding to the target identification object in the world coordinate system, the second pose data of the virtual object in the screen coordinate system of the AR device includes:

determining the first pose data according to a preset pose of the virtual object in a world coordinate system; the world coordinate system is determined according to the pre-extracted features of the target identification object;

and determining second position data of the virtual object in the screen coordinate system according to the first position data and the mapping relation between the world coordinate system and the screen coordinate system of the AR equipment.

In some embodiments, the identifying the target identification object from the current scene image includes:

extracting feature points of the current scene image to obtain feature information corresponding to at least one feature point included in the current scene image;

and comparing the characteristic information corresponding to the at least one characteristic point with the pre-stored characteristic point information of the target identification object, and identifying the target identification object in the current scene image.

In a second aspect, the disclosed embodiments provide a display apparatus for an augmented reality AR scene, the apparatus including:

the first pose determination module is configured to respond to the identification of a target identification object from a current scene image, and determine second pose data of a virtual object in a screen coordinate system of the AR device according to first pose data of the virtual object corresponding to the target identification object in a world coordinate system;

a first presentation module configured to present the virtual object at the AR device based on the second pose data;

a second presentation module configured to present the virtual object on the screen of the AR device based on preset target pose data in the screen coordinate system in response to the target identification object not being identified from the current scene image during presentation of the virtual object.

In a third aspect, an embodiment of the present disclosure provides an AR device, including:

a processor; and

a memory communicatively coupled to the processor, the memory storing computer instructions readable by the processor for causing the processor to perform the method according to any of the embodiments of the first aspect.

In a fourth aspect, the embodiments of the present disclosure provide a storage medium storing computer instructions for causing a computer to execute the method according to any one of the embodiments of the first aspect.

According to the display method of the embodiment, in the display process of the virtual object, in response to the situation that the target identification object is not identified from the current scene image, namely the target identification object Marker is lost from the current scene image, the virtual object can be displayed on the screen of the AR equipment based on the preset target pose data in the screen coordinate system, so that the display effect of the virtual object is not interrupted, and the AR display effect and the user watching experience are improved.

Drawings

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present disclosure, and other drawings can be obtained by those skilled in the art without creative efforts.

Fig. 1 is a schematic structural diagram of an AR device in accordance with some embodiments of the present disclosure.

Fig. 2 is a schematic structural diagram of an AR device in accordance with some embodiments of the present disclosure.

Fig. 3 is a block diagram of an AR device in accordance with some embodiments of the present disclosure.

Fig. 4 is a flow chart illustrating a method according to some embodiments of the present disclosure.

Fig. 5 is a schematic diagram of a demonstration method in some embodiments according to the present disclosure.

Fig. 6 is a flow chart illustrating a method according to some embodiments of the present disclosure.

Fig. 7 is a flow chart illustrating a method according to some embodiments of the present disclosure.

Fig. 8 is a schematic diagram of a demonstration method in some embodiments according to the present disclosure.

Fig. 9 is a schematic diagram of a demonstration method in some embodiments according to the present disclosure.

Fig. 10 is a schematic diagram of a demonstration method in some embodiments according to the present disclosure.

Fig. 11 is a flow chart illustrating a method according to some embodiments of the present disclosure.

Fig. 12 is a flow chart illustrating a method according to some embodiments of the present disclosure.

FIG. 13 is a block diagram of a display device according to some embodiments of the present disclosure.

Detailed Description

The technical solutions of the present disclosure will be described clearly and completely with reference to the accompanying drawings, and it is to be understood that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. All other embodiments, which can be derived by one of ordinary skill in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure. In addition, technical features involved in different embodiments of the present disclosure described below may be combined with each other as long as they do not conflict with each other.

When a virtual object is displayed in an Augmented Reality (AR) scene, a display mode of the virtual object may be determined by using a pre-made identification object (Marker). The identification object Marker is, for example, a preset pattern drawn on the surface of an article based on Hamming Code (Hamming Code) technology, and the preset pattern may be, for example, a cartoon image, a two-dimensional Code, a barcode, and the like. When the AR equipment identifies that the Marker appears in the current scene image through the camera, the corresponding virtual object can be displayed on the surface of the Marker or in the scene space where the Marker is located.

However, in the related art, during the process of displaying the virtual object, once the Marker disappears (for example, is moved away) from the current scene, the display effect of the virtual object is also interrupted, so that the virtual object cannot be displayed. That is, only when the user shoots the Marker by using the AR device, the user can view the exhibition effect of the virtual object, and in the user moving state, when the AR device does not shoot the Marker, the user expects to continue to watch the complete exhibition effect, which is not realized, resulting in poor exhibition effect.

The disclosed embodiment provides a display method and device of an Augmented Reality (AR) scene, an AR device and a storage medium, and aims to improve the display effect of a virtual object in the AR scene and improve user experience.

In some embodiments, the present disclosure provides a method for displaying an Augmented Reality (AR) scene, which may be applied to an AR device. It is to be understood that the AR device described in the embodiments of the present disclosure may be any terminal device having an AR function, for example, a head-mounted device such as AR glasses and an AR helmet, a handheld device such as a smart phone and a tablet computer, a wrist-mounted device such as a smart watch and a smart bracelet, and the present disclosure does not limit this.

As shown in fig. 1, in one example, the AR device described in the present disclosure is illustrated with AR glasses as an example. The AR device 100 includes a main body 101, and the main body 101 is a stand structure for wearing and supporting glasses. The display screen 110 is provided on the main body 101, and after the user wears the AR device 100, the display screen 110 may be positioned right in front of the eyes of the user, thereby facilitating the user to observe the picture displayed on the display screen 110. The AR device 100 further comprises a camera 120, the camera 120 being configured to capture a real scene image of the current scene. The AR device 100 further includes a processor disposed inside the main body 101, and the processor is configured to display the real scene image acquired by the camera 120 on the display screen 110 after superimposing the real scene image and the virtual object.

As shown in fig. 2, in one example, the AR device according to the present disclosure is illustrated by taking a smart phone as an example. The AR device 100 includes a housing 102, the housing 102 being a housing structure that enables the handset body to be supported. The display screen 110 is disposed on the front side of the housing 102, and the camera 120 is disposed on the front side and/or the back side of the housing 102, for example, as shown in fig. 2 (b), and the camera 120 is disposed on the back side of the housing. The camera 120 is configured to acquire a real scene image of a current scene, so that the processor of the smart phone displays the real scene image acquired by the camera 120 on the display screen 110 after superimposing the real scene image with the virtual object.

Of course, it is understood that the specific device types of the AR device are not limited to the AR glasses and the smart phone in the above examples, and the details of the disclosure are not repeated here. A block diagram of the AR device of the present disclosure is shown in fig. 3, and the AR device of the embodiment of the present disclosure is further explained with reference to fig. 3.

As shown in fig. 3, in some embodiments, AR device 100 includes a processor 130, a memory 140, a camera 120, a display screen 110, and an IMU (Inertial Measurement Unit) sensor 160.

Processor 130, memory 140, camera 120, display screen 110, and IMU sensor 160 establish a communicative coupling between any two via bus 150.

The processor 130 may be of any type, having one or more processing cores. The system can execute single-thread or multi-thread operation and is used for analyzing instructions to execute operations of acquiring data, executing logic operation functions, issuing operation processing results and the like.

The memory 140 may include a non-volatile computer-readable storage medium, such as at least one magnetic disk storage device, flash memory device, distributed storage device remotely located from the processor 130, or other non-volatile solid state storage device. The memory may have a program storage area for storing non-volatile software programs, non-volatile computer-executable programs, and modules for calling by the processor 130 to cause the processor 130 to perform one or more of the method steps described below. The memory 140 may further include a volatile random access memory medium or a storage portion such as a hard disk, which is used as a data storage area for storing the operation processing result and data issued and output by the processor 130.

The camera 120 is configured to collect a current scene image, so that the processor 130 may perform fusion processing of a virtual object according to the collected current scene image, thereby implementing an AR display effect. In the embodiment of the present disclosure, the cameras 120 may be any camera types suitable for implementation, such as a black and white camera, an RGB camera, and the like, and the number of the cameras 120 may be a monocular camera or a binocular camera, which is not limited by the present disclosure.

The display screen 110 is configured to receive the display information sent by the processor 130, so that a corresponding display image is displayed on the display screen 110 according to the display information, so that a user can observe the AR display effect through the display screen 110. In the embodiment of the present disclosure, the Display screen 110 may be any Display screen type suitable for implementation, such as an LCD (Liquid Crystal Display) Display screen, an OLED (Organic Light-Emitting semiconductor) Display screen, and the like, which is not limited in the present disclosure.

The IMU sensor 160 is an inertial measurement sensor, and is mainly used to detect and measure the acceleration and rotational motion of the AR device itself, and the basic principle thereof is to use the law of inertia, and can resolve the pose of the AR device itself in real time based on sensor signals. In some embodiments, the IMU sensors 160 may include, for example, three-axis or six-axis gyroscopes, acceleration sensors, and the like.

On the basis of the above AR device structure, a method for displaying an augmented reality AR scene according to an embodiment of the present disclosure will be described below.

As shown in fig. 4, in some embodiments, a method for presenting an augmented reality AR scene according to an example of the present disclosure includes:

s410, responding to the identification of the target identification object from the current scene image, and determining second position and posture data of the virtual object in a screen coordinate system of the AR device according to first position and posture data of the virtual object corresponding to the target identification object in a world coordinate system.

Specifically, the current scene image refers to a scene image acquired in real time by using a camera of the AR device. For example, in the AR glasses shown in fig. 1, imaging data in a viewing range is acquired by the camera 120, and an acquired scene image is displayed on the display screen 110 through data processing, where an image viewed by a user on the display screen 110 is a current scene image. It can be understood that as the pose of the AR device changes, the acquired current scene image changes, that is, the current scene image is a real-time image.

The target identification object refers to a Marker for triggering the display of the AR effect, that is, the Marker described above, and the specific presentation manner of the target identification object (Marker) may be, for example, a two-dimensional code, a preset graph, and the like, which is not limited in this disclosure.

For example, as shown in fig. 5, the target identification object 210 is a cartoon graphic drawn on the surface of the postcard 200, and the cartoon graphic may be drawn based on, for example, Hamming Code (Hamming Code) technology, so that the AR device may recognize and parse the cartoon graphic when acquiring the cartoon graphic. The process of hamming code rendering and parsing will be understood and fully implemented by those skilled in the art with reference to the relevant art, and the disclosure is not limited thereto.

For example, in another example, the target identification object is a preset pattern drawn on the surface of an object, and when a scene image including the preset pattern is acquired, the AR device may perform feature point extraction and recognition on the scene image through an image detection technology, so that the preset image may be recognized from the scene image. The image detection and preset image recognition processes can be understood and fully implemented by those skilled in the art with reference to the related art, and the present disclosure is not limited thereto.

It is to be understood that the target identification object has a corresponding virtual object, for example, in an example, a model database including at least one virtual object may be established in advance, and the virtual object in the model database has a target identification object corresponding thereto, so that when the AR device identifies the target identification object from the current scene image, the virtual object corresponding to the target identification object may be determined by searching the model database based on the correspondence relationship.

The virtual object represents an avatar rendered and presented on a display screen of the AR device, and the virtual object and a real scene image displayed on the display screen are fused and displayed, so that an interactive display effect generated by the virtual object and the real scene can be presented.

In the embodiment of the disclosure, the AR device detects and identifies the acquired current scene image through an image detection technology, and can determine the virtual object corresponding to the target identification object when the target identification object is identified from the current scene image.

When AR display is carried out based on Marker, the principle is to find the mapping relation between the pose information of the virtual object in the established world coordinate system and the pose information of the virtual object in the screen coordinate system of the AR device. In the embodiment of the disclosure, the pose information of the virtual object in the world coordinate system is defined as first pose data, and the pose information of the virtual object in the screen coordinate system of the AR device is defined as second pose data.

In some embodiments, feature extraction may be performed on the target identification object in advance based on an image detection technology to obtain at least one preset key point of the target identification object, and a world coordinate system may be established based on the at least one preset key point. After the world coordinate system is obtained, the world coordinate system and the screen coordinate system of the AR device may be aligned to obtain a mapping relationship between the two coordinate systems. When the virtual object is displayed, according to the first pose data of the virtual object in the world coordinate system, the second pose data of the virtual object in the screen coordinate system is determined through the mapping relation, and the display of the virtual object on the screen of the AR device is achieved based on the second pose data. This is specifically explained in the following embodiments of the present disclosure, and will not be described in detail here.

And S420, displaying the virtual object on the AR equipment based on the second posture data.

Specifically, after the second pose data of the virtual object in the screen coordinate system of the AR device is determined, the virtual object may be displayed and presented at the corresponding screen position according to the second pose data, so that the user may observe the virtual object at the corresponding position of the screen.

In addition, it can be understood that, because the virtual object has the above mapping relationship between the world coordinate system and the screen coordinate system, when the virtual object is fixed at a certain position of the world coordinate system, the pose of the virtual object moves along with the movement of the target identification object, and the position of the virtual object in the screen coordinate system correspondingly moves, so that the user views the visual effect that the virtual object is adsorbed on the target identification object. For example, in a real scene, when the user moves holding the postcard 200 shown in fig. 5 in his hand, a virtual object viewed on the screen of the AR device moves along with the surface of the postcard 200.

And S430, in the process of displaying the virtual object, responding to the situation that the target identification object is not identified in the current scene image, and displaying the virtual object on the screen of the AR device based on the target pose data in the preset screen coordinate system.

In an example scenario, taking the postcard 200 shown in fig. 5 as an example, when the user recognizes the target identification object 210 on the postcard 200 through the AR device, the display effect of the corresponding virtual object may be viewed on the display screen of the AR device.

When the relative position of the postcard 200 and the AR device 100 changes, for example, the user moves the postcard 200 out of the viewing range of the AR device 100, that is, the camera 120 of the AR device 100 cannot recognize the target identification object 210, the display effect of the virtual object is not directly interrupted at this time, but the virtual object continues to be displayed based on the preset target pose data.

Specifically, target pose data representing data on the presentation position and posture of the virtual object in the screen coordinate system may be set in advance in the screen coordinate system of the AR device. Thus, when the AR device 100 cannot recognize the target identification object, the display effect of the virtual object is not interrupted, but the virtual object continues to be displayed with the preset target pose data.

In some embodiments, the target pose data comprises presentation position data for the virtual object. For example, in one example, the presentation location data may represent a virtual object presented in the center of the display screen of the AR device. For example, in another example, the presentation location data may represent a virtual object presented in a lower right corner of a display screen of the AR device. For example, in yet another example, the presentation position data of the virtual object on the display screen of the AR device may be located in real-time using Simultaneous Localization And Mapping (SLAM) technology based on the initial position of the virtual object. The present disclosure is not so limited.

In some embodiments, the target pose data includes presentation pose data for the virtual object, e.g., the presentation pose data may represent a rotation of the virtual object at the presentation position; for another example, the display posture data may indicate that the virtual object performs a preset action at the display position; for another example, the display pose data may be the same as the pose data included in the second pose data, i.e., the virtual object continues to be displayed in the previous pose; etc., to which the present disclosure is not limited.

When the virtual object is displayed, a display position of the virtual object on the screen of the AR device may be determined based on the display position data, and a display posture of the virtual object may be determined based on the display posture data, so that the virtual object is displayed at the corresponding display position in the display posture.

In some embodiments, when the target identification object reappears in the current scene image, the AR device may reappear the AR effect based on the manner described in the foregoing S410 and S420. This is explained in the following embodiments of the present disclosure, and will not be described in detail here.

According to the display method provided by the embodiment of the disclosure, in the AR effect display scene, under the condition that the Marker cannot be identified in the current scene image, the display of the virtual object cannot be interrupted, so that the position of using the AR equipment by a user is not limited, and the user can be given higher degree of freedom.

For example, in an example scenario, when a user wears the AR glasses shown in fig. 1 to visit a scene in which target identification objects are arranged, the user desires to view a complete presentation corresponding to a certain target identification object. At the moment, the user does not need to be limited to stay at the position of the target identification object, the user can continue to move in the virtual object display process, and even if the target identification object cannot be acquired by the AR glasses, the complete content can be displayed on the display screen, so that the user can view the complete displayed content, and the user experience is improved.

Therefore, in the AR display scene, the display of the virtual object cannot be interrupted under the condition that the Marker cannot be identified in the current scene image, the AR display effect is improved, and the user experience is improved.

In the embodiment of the disclosure, when the AR effect is displayed, the AR device needs to recognize a target identification object (Marker) from the collected current scene image. As shown in fig. 6, in some embodiments, the process of identifying a target Marker object (Marker) from a current scene image includes:

s610, extracting the feature points of the current scene image to obtain feature information corresponding to at least one feature point included in the current scene image.

S620, comparing the characteristic information corresponding to the at least one characteristic point with the characteristic point information of the target identification object stored in advance, and identifying the target identification object in the current scene image.

Specifically, the AR device 100 may acquire the current scene image through the camera 120, and it is understood that the current scene image may be a single frame image acquired by the camera or a multi-frame image in a video stream acquired by the camera, which is not limited in this disclosure.

In the process of processing the current scene image, a detection area including the object to be detected in the current scene image can be located through an image detection algorithm, and then one or more feature points of each object to be detected are extracted and obtained in the detection area.

Taking fig. 5 as an example of the current scene image, the image detection algorithm may be used to first determine that the area of the postcard 200 is the area to be detected, and then perform feature point extraction on the "small dragon" and the "lucky cloud" waiting object included in the postcard 200 to obtain feature information of each object to be detected.

After the feature information of each object to be detected is obtained, the feature information of the object to be detected can be compared with the feature point information of the target identification object stored in advance, and if the similarity between the feature point information and the feature point information is greater than a preset threshold value, the object to be detected is the target identification object; otherwise, if the similarity is not greater than the preset threshold, it indicates that the target identification object is not included in the current scene image.

Still taking fig. 5 as an example, it is assumed that the feature point information of the target identification object stored in advance corresponds to a cartoon image of a "little dragon", so that the feature information of the little dragon extracted from the current scene image is compared with the feature point information stored in advance, and the similarity between the two is greater than a preset threshold value, so as to determine that the target identification object is identified in the current scene image.

For example, the information of the feature point according to the embodiment of the present disclosure may include a texture feature value, an RGB feature value, a gray scale value, and the like corresponding to the feature point, which can identify the feature of the feature point, and the present disclosure does not limit this.

When the target identification object is identified from the current scene image, the virtual object corresponding to the target identification object, the pose data required by the virtual object display and the like can be further acquired. The following description is made with reference to the embodiment of fig. 7.

As shown in fig. 7, in some embodiments, the presentation method of the examples of the present disclosure, the process of determining the second pose data of the virtual object includes:

and S710, determining first pose data according to the preset pose of the virtual object in the world coordinate system.

S720, according to the first position data and the mapping relation between the world coordinate system and the screen coordinate system of the AR device, determining second position data of the virtual object in the screen coordinate system.

Specifically, taking fig. 5 as an example of the current scene image, the target identification object is a cartoon image drawn on the surface of the postcard 200, and when the AR device detects and identifies the target identification object included in the current scene image through the foregoing embodiment of fig. 6, feature extraction may be performed on the target identification object based on an image detection technology to obtain at least one preset key point of the target identification object, and then a world coordinate system is established according to the extracted preset key point.

For example, as shown in fig. 5, the preset key point may be a center point of the target identification object, the center point of the target identification object is taken as an origin O of a coordinate system, a direction parallel to a short side of the postcard 200 and passing through the origin O is taken as an X-axis, a direction parallel to a long side of the postcard 200 and passing through the origin O is taken as a Y-axis, and a direction perpendicular to the X-axis and the Y-axis and passing through the origin O is taken as a Z-axis, thereby establishing a world coordinate system O-XYZ.

It is understood that the process of establishing the world coordinate system in fig. 5 is only an example of the embodiment of the present disclosure, and in other embodiments, the world coordinate system may also be established according to other preset key points, for example, the coordinate system may be established by using the vertex of the lower left corner of the postcard 200 as the coordinate origin O, which is not limited by the present disclosure.

With continued reference to fig. 5, after the world coordinate system is established, the world coordinate system may be aligned with the screen coordinate system of the AR device, thereby establishing a mapping relationship between the world coordinate system and the screen coordinate system. For example, in some embodiments, the world coordinate system may be mapped to the camera coordinate system of the AR device through a rotation and translation process and then mapped from the camera coordinate system into the screen coordinate system of the AR device.

For the alignment process of the world coordinate system and the screen coordinate system, those skilled in the art can understand and fully implement the alignment process with reference to the related art, and the detailed description of the disclosure is omitted.

Based on the foregoing, when the AR effect is displayed, the preset display effect includes related data required for displaying the virtual object, such as model rendering data of the virtual object and pose data of the virtual object. For example, in the example of fig. 5, the preset display effect is: the cartoon image "little dragon" rotates at the position shown in the figure. Therefore, the display effect data comprises model rendering data required by rendering the cartoon image and pose data representing the rotation of the cartoon image at a preset position.

Therefore, the AR equipment can determine the first posture data of the virtual object in the world coordinate system according to the preset display effect data. Taking the virtual object "dragon" as an example, the exhibition effect data indicates that the virtual object continuously rotates at the position of the origin O of the world coordinate system, so that the first attitude data determined by the AR device according to the exhibition effect data, that is, the position data including the virtual object, is the origin O (0,0,0), and the attitude data is rotation-related data.

After the first position data is determined, the AR device may map the first position data of the virtual object in the world coordinate system to the screen coordinate system according to the mapping relationship, so as to obtain the second position data of the virtual object in the screen coordinate system. After determining the second pose data, the AR device may display the virtual object under the display screen according to the second pose data. In the example of fig. 5, the AR effect exhibited on the AR device may be as shown with reference to fig. 8, with the virtual object presenting a avatar on the surface of the postcard 200 and the virtual object presenting a rotated effect at the corresponding location.

As shown in fig. 8, the display position of the virtual object "dragon" on the display screen of the AR device is determined based on the mapping of the pose data in the world coordinate system to the screen coordinate system. Therefore, when the position and posture of the postcard 200 change, the established world coordinate system changes, the position and posture of the virtual object changes along with the change of the coordinate system, and the corresponding display effect shows that the virtual object is adsorbed on the surface of the postcard 200, and the position and posture of the virtual object changes along with the change of the postcard 200.

In some example scenarios, the AR device location remains unchanged and the user moves the postcard 200 shown in fig. 5 to the left in the real world until moving out of the viewing range of the camera of the AR device. In the moving process, a current scene image acquired by the AR device at a certain moment is as shown in fig. 9, at this time, the incomplete target identification object 210 cannot be recognized or can only be recognized in the current scene image, and in the embodiment of the present disclosure, the virtual object is displayed on the display screen of the AR device according to the preset target pose data based on the method flow described in the foregoing fig. 4.

In fig. 10, a presentation effect on a display screen of an AR device according to a presentation method of an embodiment of the present disclosure is shown. As shown in fig. 10, in this example, in a case where the target identification object cannot be recognized in the current scene image, the virtual object is presented in the center of the display interface of the display screen 110.

Of course, it is understood that fig. 10 is only an exemplary illustration, and in other embodiments, any other illustration suitable for implementation may be implemented by those skilled in the art based on the inventive concept of the present disclosure, and the disclosure is not enumerated herein.

In some embodiments, considering that in some scenes, the AR display effect has coherence, in order to ensure that the user does not interrupt viewing, the continuity of the AR display content can be maintained before and after the target identification object cannot be identified in the current scene image. The following description will be made with reference to the embodiment of fig. 11.

As shown in fig. 11, in some embodiments, the demonstration method of the present disclosure further includes:

s1110, in the process of displaying the virtual object, in response to the fact that the target identification object is not recognized from the current scene image, determining the current displayed progress.

And S1120, continuously displaying the virtual object on the screen of the AR device based on the target pose data and the displayed progress.

Specifically, in the process of displaying the virtual object, the displayed content often has a display progress. For example, for a cultural relic exhibition hall scene, after a user scans a Marker arranged in the scene in advance, the content displayed by the AR device can be a video or animation of a cultural relic introduction.

In this example, when a target identification object (Marker) cannot be identified in a current scene image, the playing progress of a currently displayed video or animation may be marked, so that by the display method described in the above embodiment, a virtual object is continuously displayed at a preset position of a display screen of the AR device at the current playing progress, that is, continuous playing is realized, and continuity of displayed content before and after the Marker disappears is ensured.

In some embodiments, the display content may further include audio data, and the audio data may also be played continuously based on the foregoing manner, so that the picture content or the audio content seen by the user is consistent before and after the Marker disappears, thereby improving the AR display effect.

By the aid of the method, the virtual object is continuously displayed on the screen of the AR equipment according to the displayed progress, continuity of displayed contents before and after Marker disappears is guaranteed, and AR display effect and user experience are improved.

In the above embodiment, in the case that the target identification object is not identified in the current scene image, the uninterrupted display of the virtual object may be realized according to the above embodiment process. In some embodiments, when the target identification object reappears in the current scene image, the processes of the above embodiments may also be re-executed, so that the virtual object reappears the display effect adsorbed on the target identification object.

For example, in some embodiments, the display method of the present disclosure further includes:

responsive to re-identifying the target identifying object from the current scene image, re-performing the step of presenting the virtual object at the AR device based on the second pose data.

For example, in the example of fig. 10, the target identification object cannot be identified or cannot be identified completely in the current scene image, so that the virtual object is presented at the preset position on the display screen 110 of the AR device. In one example scenario, in a real scene, the position of the AR device remains unchanged, and the postcard 200 gradually moves to the right, so that the target identification object is re-identified in the current scene image captured by the camera of the AR device.

In this case, based on the process described in the foregoing embodiment of fig. 4, the steps S410 to S420 may be re-executed, so that the virtual object is re-displayed on the surface of the postcard based on the second posture data, and appears as a display effect attached to the surface of the postcard. For a specific process, those skilled in the art can understand and fully implement the method by referring to the foregoing embodiments, and the detailed description of the disclosure is omitted.

Therefore, in the embodiment of the disclosure, under the condition that the target identification object is re-identified in the current scene image, the second pose data can be determined based on the target identification object again to display the virtual object, so that the interaction effect is enhanced, and the display effect and the user experience are improved.

In some embodiments, to enhance the AR display effect, different display effects may be rendered for the virtual object for both the case where the target identification object is recognized and the case where the target identification object is not recognized. The following description will be made with reference to the embodiment of fig. 12.

As shown in fig. 12, in some embodiments, the demonstration method of the examples of the present disclosure further includes:

s1210, under the condition that the target identification object is identified from the current scene image, the virtual object is displayed with a first display effect.

And S1220, under the condition that the target identification object is not recognized from the current scene image, displaying the virtual object with a second display effect.

Specifically, the first presentation effect is different from the second presentation effect. It is to be understood that the presentation effect may include multiple dimensions, such as brightness, transparency, degree of blurring, size, presentation position, presentation angle, etc., and that the first presentation effect and the second presentation effect may differ in at least one dimension.

For example, in a case that the target identification object can be recognized in the current scene image, the display effect of the virtual object may be as shown in fig. 8, that is, the first display effect of the virtual object includes, for example, brightness, transparency, blurring degree, position, and size, which are all displayed according to the second pose data.

And under the condition that the target identification object cannot be identified in the current scene image, the virtual object is not displayed with the first display effect any more, but is displayed with the second display effect. For example, the transparency of the virtual object is increased, from opaque to translucent; for another example, the size of the virtual object is reduced from the original size to half; for another example, the brightness of the virtual object becomes dark, and the original brightness becomes half brightness; etc., which are not enumerated by this disclosure.

In an example scenario, when a user wears the AR glasses shown in fig. 1 to visit an exhibition hall in which a target marker is pre-arranged, as the AR glasses recognize the target marker object in the current scene image, the virtual object may be displayed in an opaque manner according to the foregoing embodiment. As the user position moves, the target identification object cannot be captured by the AR glasses, thereby presenting the virtual object with 50% transparency on the display screen of the AR glasses. When the target identification object is collected again by the AR glasses, the virtual object is displayed on the display screen of the AR glasses in an opaque mode again.

The above is merely an example of the embodiment of the disclosure, and in other embodiments, the exhibition effect may have other differences before and after the Marker disappears, which is not limited by the disclosure.

By the method, different virtual object display effects are set before and after Marker disappears in the embodiment of the disclosure, so that on one hand, interactivity and interestingness are enhanced, on the other hand, the user target identification object is prompted to disappear from the current scene image, and user experience is improved.

The embodiment of the disclosure provides a display device for an Augmented Reality (AR) scene, which can be applied to AR equipment. In some embodiments, the AR device may be, for example, the AR glasses shown in fig. 1. In other embodiments, the AR device may be a smartphone, such as that shown in fig. 2. The present disclosure is not so limited.

As shown in fig. 13, in some embodiments, a presentation apparatus of an augmented reality AR scene of an example of the present disclosure includes:

a first pose determination module 10 configured to, in response to identifying the target identification object from the current scene image, determine second pose data of the virtual object in a screen coordinate system of the AR device according to first pose data of the virtual object corresponding to the target identification object in a world coordinate system;

a first presentation module 20 configured to present the virtual object at the AR device based on the second pose data;

and the second presentation module 30 is configured to, in response to that the target identification object is not recognized from the current scene image, present the virtual object on the screen of the AR device based on preset target pose data in the screen coordinate system during presentation of the virtual object.

In some embodiments, second display module 30 is configured to:

in the display process of the virtual object, in response to the fact that the target identification object is not recognized from the current scene image, determining the current displayed progress;

and continuously displaying the virtual object on the screen of the AR device based on the target pose data and the displayed progress.

In some embodiments, second display module 30 is configured to:

In some embodiments, first display module 20 is configured to: under the condition that a target identification object is identified from a current scene image, displaying a virtual object with a first display effect;

second display module 30 is configured to: and in the case that the target identification object is not recognized from the current scene image, displaying the virtual object with a second display effect, wherein the second display effect is different from the first display effect.

In some embodiments, the target pose data comprises presentation position data and presentation pose data for the virtual object; second display module 30 is configured to:

at the screen position indicated by the presentation position data, the virtual object is presented in the pose indicated by the presentation pose data.

In some embodiments, the first pose determination module 10 is configured to:

determining first pose data according to a preset pose of the virtual object in a world coordinate system; the world coordinate system is determined according to the characteristics of the target identification object extracted in advance;

In some embodiments, the first pose determination module 10 is configured to:

and comparing the characteristic information corresponding to the at least one characteristic point with the characteristic point information of the target identification object stored in advance, and identifying the target identification object in the current scene image.

Therefore, in the AR display scene, the display of the virtual object cannot be interrupted under the condition that the Marker cannot be identified in the current scene image, the AR display effect is improved, and the user experience is improved. Different virtual object display effects are set before and after Marker disappears, so that on one hand, interactivity and interestingness are enhanced, on the other hand, a user target identification object is prompted to disappear from a current scene image, and user experience is improved.

The disclosed embodiments provide an AR device, comprising:

a processor; and

a memory communicatively coupled to the processor, the memory storing computer instructions readable by the processor for causing the processor to perform a method according to any of the embodiments of the first aspect.

The disclosed embodiments provide a storage medium storing computer instructions for causing a computer to perform a method according to any one of the embodiments of the first aspect.

In particular, for the AR device and storage medium related implementations of the examples of the present disclosure, those skilled in the art can understand and fully implement the embodiments with reference to the foregoing description, and the detailed description of the present disclosure is omitted here.

The disclosure relates to the field of augmented reality, and aims to detect or identify relevant features, states and attributes of a target object by means of various visual correlation algorithms by acquiring image information of the target object in a real environment, so as to obtain an AR effect combining virtual and reality matched with specific applications. For example, the target object may relate to a face, a limb, a gesture, an action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, a display area, a display item, etc. associated with a venue or a place. The vision-related algorithms may involve visual localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, and the like. The specific application can not only relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also relate to special effect treatment related to people, such as interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like. The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through the convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

It should be understood that the above embodiments are only examples for clearly illustrating the present invention, and are not intended to limit the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications of the present disclosure may be made without departing from the scope of the present disclosure.

Claims

1. A display method of an Augmented Reality (AR) scene, the method comprising:

presenting the virtual object at the AR device based on the second pose data;

2. The method according to claim 1, wherein the presenting the virtual object on the screen of the AR device based on preset target pose data in the screen coordinate system in response to the target identification object not being identified from the current scene image during the presenting of the virtual object comprises:

3. The display method according to claim 1 or 2, further comprising:

4. The display method according to any one of claims 1 to 3, further comprising:

5. The method of displaying of any one of claims 1 to 4, wherein the object pose data comprises display position data and display pose data for the virtual object; the displaying the virtual object on the screen of the AR device based on preset target pose data in the screen coordinate system comprises:

6. The presentation method according to any one of claims 1 to 5, wherein the determining, according to the first pose data of the virtual object corresponding to the target identification object in the world coordinate system, the second pose data of the virtual object in the screen coordinate system of the AR device includes:

7. The method as claimed in any one of claims 1 to 5, wherein the identifying a target identification object from the current scene image comprises:

8. A presentation apparatus of an Augmented Reality (AR) scene, the apparatus comprising:

9. An AR device, comprising:

a processor; and

a memory communicatively coupled with the processor, the memory storing computer instructions readable by the processor for causing the processor to perform the method of any of claims 1 to 7.

10. A storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1 to 7.