JP2023504608A - Display method, device, device, medium and program in augmented reality scene - Google Patents

Abstract

Embodiments of the present application provide a display method, apparatus, apparatus, medium and program in an augmented reality scene. The method is performed by an electronic device, and the display method includes obtaining a current scene image taken by an AR device; determining first display position data of an AR special effect that matches a physical object, and controlling the AR device to display the AR special effect based on the first display position data; and the AR special effect. in the process of displaying, in response to the target physical object not being identified in the current scene image, determining second display position data for the AR special effect; based on the second display position data; controlling the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect.

Description

(Cross reference to related applications)
This application is filed on November 06, 2020, with the application number 202011232913.8, the applicant being "Beijing SenseTime Technology Development Company, Limited", and the invention title being "A display method in an augmented reality scene, Devices, Electronics and Storage Media”, the entire content of which is incorporated herein by reference.

TECHNICAL FIELD The present application relates to the field of augmented reality technology, and more particularly to a display method, device, device, medium, and program in an augmented reality scene.

Augmented Reality (AR) technology simulates entity information (vision information, voice, touch, etc.) and then superimposes it on the real world to create a real environment and virtual objects on the same screen or space in real time. Present.

In the related art, with the development of AR technology, the application area of AR equipment has become more and more wide. AR devices can display AR special effects that are superimposed on a physical object. In the process of displaying AR special effects, it is generally necessary to determine the display position of the AR special effects. In some cases, the physical object or AR device may move. How to continue to determine the display position of the AR special effect when the position of the physical object changes during the movement process, so as to consistently display the AR special effect and provide a more realistic display effect. is an issue worthy of research.

Embodiments of the present application provide a display scheme in an augmented reality scene.

Embodiments of the present application provide a display method in an augmented reality scene. The method is performed by an electronic device, the method comprising:
obtaining a current scene image captured by an AR device;
determining first display position data of an AR special effect that matches the target physical object if the current scene image includes a target physical object; and based on the first display position data, the AR controlling the AR device to display a special effect;
In the process of displaying the AR special effect, determining second display position data of the AR special effect in response to the target physical object not being identified in the current scene image; and controlling the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect based on.

In this way, the identification result of the target entity object can trigger the display of matching special effect data, the display effect being closely associated with the target entity object, and the special effect data being more closely associated with the target entity object. can be displayed in relation to each other. And then, according to the identification result of the target entity object, control the AR equipment to display the AR special effects with different positioning methods, so as to improve the consistency and stability of the AR special effects in the display process, and the AR The display of special effects can be made more realistic.

In some embodiments of the present application, if it is identified that the current scene image includes a target entity object, before determining first display position data for an AR special effect that matches the target entity object, said performing feature point extraction on a current scene image, obtaining feature information corresponding to each of a plurality of feature points included in the current scene image, and detecting the plurality of feature points in a target detection region in the current scene image; based on the feature information corresponding to each of the plurality of feature points and the feature information corresponding to each of the plurality of feature points included in the target entity object stored in advance, and comparing the current scene image with the feature information corresponding to each of the plurality of feature points. Identifying whether the target physical object is included in the current scene image by determining whether the target physical object is included. Thus, by extracting a plurality of feature points included in the target detection area, it is possible to identify whether or not the target entity object is included in the current scene image. It can quickly and accurately determine whether an entity object is included.

In some embodiments of the present application, determining first display position data of an AR special effect that matches the target physical object comprises obtaining position information of the target physical object in the current scene image; determining position data of the target physical object in a pre-established world coordinate system based on position information, and determining position data of the AR device in the world coordinate system based on the current scene image; determining the first display position data based on position data of the target physical object in the world coordinate system and position data of the AR device in the world coordinate system. Thus, by determining the position data of the target entity object and the AR device in the same world coordinate system, the first display position data of the AR special effect for the AR device in the same world coordinate system can be determined more accurately. , thus making the augmented reality scene displayed by the AR device more realistic.

In some embodiments of the present application, determining position data of the target physical object in a pre-established world coordinate system based on the position information corresponds to the position information, the image coordinate system and the AR device. determining the position data of the target entity object in the world coordinate system based on the transformation relationship with the camera coordinate system and the transformation relationship between the camera coordinate system corresponding to the AR device and the world coordinate system; Determining the first display position data based on the position data of the target physical object in the world coordinate system and the position data of the AR device in the world coordinate system includes determining the position of the target physical object in the world coordinate system. determining position data of the AR special effect in the world coordinate system based on the data; and based on position data of the AR special effect in the world coordinate system and position data of the AR device in the world coordinate system. and determining the first display position data. Thus, when it is identified that the current scene image includes the target entity object, the position data of the target entity object and the AR device in the world coordinate system can be accurately determined based on the current scene image, Thereby, the first display position data of the AR special effect can be obtained accurately and quickly.

In some embodiments of the present application, determining the second display position data of the AR special effect includes: determining relative position data between the AR device and the target physical object when taking a current scene image based on the relative position data between the AR device and the target physical object; and determining second display position data for the AR special effect based on. In this way, using the relative position data of the current scene image, the past scene image, and the AR device and the target entity object in the world coordinate system when the past scene image is taken, the AR device and the target entity object when the current scene image is taken The relative position data with the target physical object can be determined relatively accurately, and thus the second display position data of the AR special effect can be determined based on the accurate relative position data; It can be realized that the AR special effects can be displayed even if the target physical object has not been identified.

In some embodiments of the present application, the AR special effect includes an AR screen and audio content matching the AR screen; Controlling the AR device to continue to display a special effect includes displaying the second display if the target entity object has not been identified in the current scene image and the AR screen display has not been completed. Based on the location data, controlling the AR device to keep displaying audio content matching the AR screen that is not being displayed according to the displayed progress of the AR screen. In this way, when the target entity object is not identified, if the AR screen display is not completed, it can also continue to display the audio content matching the AR screen, improving the consistency of the AR special effect display. , AR special effects can be displayed more realistically.

In some embodiments of the present application, in the process of displaying the AR special effect, in response to the target physical object not being identified in the current scene image, displaying second display position data of the AR special effect. After determining and controlling the AR device to continue to display the AR special effect according to the displayed progress of the AR special effect based on the second display position data, the method performs the Controlling the AR device to continue to display the AR special effect according to the displayed progress of the AR special effect again based on the determined first display position data when the target entity object is re-identified. further includes In this way, after the target entity object is re-identified, the AR special effect is continued to be displayed again based on the first display position data with relatively high accuracy, thereby improving the consistency of the AR special effect in the display process. And stability can be improved, and the display of AR special effects can be more realistic.

In some embodiments of the present application, before controlling the AR device to display the AR special effect, the method comprises obtaining an AR special effect that matches the target physical object, The AR special effect includes special effect data corresponding to each of the plurality of virtual objects, and controlling the AR device to display the AR special effect comprises: and controlling the AR device to sequentially display the special effect data corresponding to each of the plurality of virtual objects according to the display order of the special effect data corresponding to the virtual objects. In this way, an AR special effect composed of multiple virtual objects can be displayed to the user, making the display of the AR special effect more lively.

In some embodiments of the present application, the target physical object includes a calendar, and before controlling the AR device to display the AR special effect, the method obtains an AR special effect that matches the calendar. wherein the AR special effect includes first special effect data generated based on the cover content of the calendar, and controlling the AR device to display the AR special effect. and controlling the AR device to display the AR special effect matching the calendar cover content based on the first special effect data when the calendar cover content is identified. include. In this way, when the calendar cover is identified, an AR special effect introducing the calendar can be displayed on the calendar cover, enriching the display contents of the calendar and enhancing the user's viewing interest in the calendar.

In some embodiments of the present application, the target physical object includes a calendar, and before controlling the AR device to display the AR special effect, the method obtains an AR special effect that matches the calendar. wherein the AR special effects include second special effect data generated based on mark events at the same time in the past on at least one predetermined date in the calendar; Controlling the AR device to display the AR special effect includes performing at least one predetermined date in the calendar based on the second special effect data when at least one predetermined date in the calendar is identified. and controlling the AR device to display the AR special effect that matches the date of the date. In this way, when the calendar is displayed to the user and at the same time a predetermined date in the calendar is acquired, the AR special effect corresponding to the predetermined date is displayed to the user to enrich the display contents of the calendar. can also

For the description of the effects of the following devices, electronic devices, etc., refer to the description of the display method in the augmented reality scene.

Embodiments of the present application provide a display device in an augmented reality scene. The device comprises:
an acquisition module configured to acquire a current scene image taken by the AR device;
determining first display position data of an AR special effect that matches the target physical object if the current scene image includes a target physical object; and based on the first display position data, the AR a first control module configured to control the AR device to display a special effect;
In the process of displaying the AR special effect, determining second display position data of the AR special effect in response to the target physical object not being identified in the current scene image; and a second control module configured to control the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect based on.

In some embodiments of the present application, the acquisition module performs feature point extraction on the current scene image to obtain feature information corresponding to each of a plurality of feature points included in the current scene image, and A plurality of feature points are located in a target detection area in the current scene image, and correspond to feature information corresponding to each of the plurality of feature points and feature points included in the target entity object stored in advance. determining whether the current scene image includes the target entity object by performing a comparison based on the feature information to determine whether the current scene image includes the target entity object. Configured.

In some embodiments of the present application, the first control module obtains location information of the target entity object in the current scene image, and based on the location information, locates the target entity in a pre-established world coordinate system. determining position data of an object; determining position data of the AR device in the world coordinate system based on the current scene image; It is configured to determine the first display position data based on position data of the AR device.

In some embodiments of the present application, the first control module controls the position information, the transformation relationship between the image coordinate system and the camera coordinate system corresponding to the AR device, and the camera coordinate system corresponding to the AR device and the world coordinates. The first control module is configured to determine position data of the target physical object in the world coordinate system based on a transformation relationship with a system, wherein the first control module is configured to determine position data of the target physical object in the world coordinate system. to determine the position data of the AR special effect in the world coordinate system, and based on the position data of the AR special effect in the world coordinate system and the position data of the AR device in the world coordinate system, the first display configured to determine position data;

In some embodiments of the present application, the second control module aligns the AR device and the target entity object in a pre-established world coordinate system when taking the current scene image, the past scene image, and the past scene image. determining the relative position data between the AR device and the target physical object when taking the current scene image based on the relative position data of the second relative position data of the AR special effect based on the relative position data configured to determine display position data;

In some embodiments of the present application, the AR special effect includes an AR screen and audio content matching the AR screen, and the second control module determines whether the target physical object has been identified in the current scene image. and the display of the AR screen is not completed, continue to display audio content matching the AR screen that is not displayed according to the display progress of the AR screen based on the second display position data. to control the AR device.

In some embodiments of the present application, when the target entity object is re-identified in the current scene image, the first control module re-executes the AR special display based on the determined first display position data. It is configured to control the AR device to keep displaying the AR special effect according to the displayed progress of the effect.

In some embodiments of the present application, before the first control module is configured to control the AR device to display the AR special effect, the acquisition module matches the target physical object. wherein the AR special effect includes special effect data corresponding to each of a plurality of virtual objects; and the first control module is configured to obtain an AR special effect corresponding to each of the plurality of virtual objects. The AR device is configured to sequentially display the special effect data corresponding to each of the plurality of virtual objects according to the display order of the special effect data.

In some embodiments of the present application, the target physical object includes a calendar, and before the first control module is configured to control the AR device to display the AR special effect, the obtaining a module configured to obtain an AR special effect matching the calendar, the AR special effect including first special effect data generated based on the cover content of the calendar, the first control module comprising: and, if the calendar cover content is identified, control the AR device to display the AR special effect matching the calendar cover content based on the first special effect data. .

In some embodiments of the present application, the target physical object includes a calendar, and before the first control module is configured to control the AR device to display the AR special effect, the obtaining The module is configured to obtain an AR special effect matching the calendar, wherein the AR special effect includes a first event generated based on mark events at the same time in the past of at least one predetermined date in the calendar. two special effect data are included, and the first control module determines, based on the second special effect data, at least one predetermined date in the calendar if at least one predetermined date in the calendar is identified; It is configured to control the AR device to display the matching AR special effect.

Embodiments of the present application further provide an electronic device. The electronic device comprises a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device operates, the processor and the memory , communicates over a bus and, when the machine-readable instructions are executed by the processor, perform a method of display in an augmented reality scene according to any one embodiment.

Embodiments of the present application further provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium and causes a processor to perform a method of displaying in an augmented reality scene according to any one embodiment.

Embodiments of the present application further provide computer programs. The computer program includes computer readable code that causes a processor of an electronic device to perform a method of displaying in an augmented reality scene as described in any one of the embodiments above.

Embodiments of the present application provide at least a display method, apparatus, apparatus, medium and program in an augmented reality scene, which can be triggered to display matching special effect data according to the identification result of the target entity object, and the display thereof. Effects are closely associated with the target physical object, allowing special effects data to be displayed more relevantly to the target physical object. And then, according to the identification result of the target entity object, control the AR equipment to display the AR special effects with different positioning methods, so as to improve the consistency and stability of the AR special effects in the display process, and the AR The display of special effects can be made more realistic.

In order to make the above objects, features and advantages of the present application clearer and easier to understand, preferred embodiments are described in detail below with reference to the accompanying drawings.

Fig. 3 shows a flow chart of a display method in an augmented reality scene according to an embodiment of the present application; Fig. 2 shows a schematic diagram of a system architecture in which the display method in an augmented reality scene of an embodiment of the present application can be applied; FIG. 4 shows a flow chart of a method for determining whether a current scene image includes a target physical object according to an embodiment of the present application; FIG. 4 shows a flowchart of a method for determining display position data for AR special effects according to an embodiment of the present application; FIG. 4 illustrates a flow chart of another method for determining display position data for AR special effects according to an embodiment of the present application; FIG. FIG. 4 illustrates a flow chart of another method for determining display position data for AR special effects according to an embodiment of the present application; FIG. FIG. 4 shows a schematic diagram of a display screen of an AR special effect according to an embodiment of the present application; Fig. 8 shows a structural schematic diagram of a display device 800 in an augmented reality scene according to an embodiment of the present application; FIG. 9 shows a structural schematic diagram of an electronic device 900 according to an embodiment of the present application;

In order to describe the technical solutions in the embodiments of the present application more clearly, the drawings required in the embodiments are briefly described below. The drawings attached hereto are incorporated into the specification and constitute a part of the specification, and the drawings show the embodiments compatible with the present application, and together with the description, interpret the technical solution of the present application. used for It should be understood that the following drawings depict several embodiments of the present application for purposes of illustration only and are not to be considered limiting in scope. Those skilled in the art can also derive other related drawings based on these drawings without creative effort.

In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following clearly and completely describes the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. . Of course, the described embodiments are only some embodiments of the present application rather than all embodiments. Generally, the components of the embodiments of the present application illustrated and illustrated in the drawings herein can be arranged and designed in a wide variety of different configurations. Accordingly, the following detailed description of embodiments of the present application provided in the drawings is intended only to illustrate selected embodiments of the present application and not to limit the scope of the claimed application. Based on the embodiments of the present application, all other embodiments obtained by persons skilled in the art without creative efforts fall within the scope of protection of the present application.

In the following drawings, similar symbols and letters represent similar elements, so that once an element is defined in one drawing, it need not be defined and explained in subsequent drawings. Please note.

As used herein, the term "and/or" is used to describe a related relationship, indicating that there are three possible relationships. For example, A and/or B represents three cases: only A is present, A and B are present at the same time, and only B is present. Also, as used herein, the term "at least one" represents any one of the plurality or any combination of at least two of the plurality. For example, including at least one of A, B, and C means including any one or more elements selected from the set consisting of A, B, and C.

A plurality or a plurality of types in the examples of this application may refer to at least two or at least two types, respectively.

With the development of AR technology, AR technology has been gradually applied to multiple fields. For example, an AR special effect can be superimposed on a physical object, and the AR special effect can introduce the physical object to the user in a lively manner. In the process of displaying AR special effects, it is generally necessary to determine the display position of the AR special effects. In the process of displaying AR special effects to the user, the related art may move the physical object or the AR device. How to continue to determine the display position of the AR special effect when the position of the physical object changes during the movement process, so as to consistently display the AR special effect and provide a more realistic display effect. is an issue worthy of research.

According to the above research, the embodiments of the present application provide a display method, apparatus, equipment, medium and program in an augmented reality scene, which can be triggered to display matching special effect data according to the identification result of the target entity object. , the display effect can be closely associated with the target physical object, and the special effect data can be displayed in a more relevant manner with the target physical object. And then, according to the identification result of the target entity object, control the AR equipment to display the AR special effects with different positioning methods, so as to improve the consistency and stability of the AR special effects in the display process, and the AR The display of special effects can be made more realistic.

For easy understanding of the embodiments of the present application, first, the display method in the augmented reality scene disclosed by the embodiments of the present application will be described in detail. The execution subject of the display method in an augmented reality scene according to the embodiments of the present application is generally a computer device with some computing power. Such computer equipment includes, for example, terminal equipment, servers or other processing equipment. The terminal device may be an AR device with an AR function, such as AR glasses, tablets, smart phones, smart wearable devices, etc., which have display functions and data processing capabilities. Not limited. In some embodiments of the present application, the method of displaying in the augmented reality scene may be implemented by a processor invoking computer readable instructions stored in memory.

Referring to FIG. 1, FIG. 1 is a flow chart of a display method in an augmented reality scene according to an embodiment of the present application. The display method includes the following S101 to S103.

At S101, a current scene image captured by an AR device is obtained.

Exemplary AR devices may include, but are not limited to, devices with display capabilities and data processing capabilities, such as AR glasses, tablets, smartphones, smart wearable devices, and the like. An application program for displaying AR scene content may be installed on these AR devices, and the user can experience the AR scene content in the application program.

Illustratively, the AR device may include an image gathering member, eg, a Red Green Blue (RGB) camera, for taking images. After obtaining the current scene image taken by the AR device, identification can be performed on the current scene image to identify whether it contains a target entity object that triggers the display of special effect data.

In S102, if it is identified that the target entity object is included in the current scene image, determine the first display position data of the AR special effect that matches the target entity object, and based on the first display position data, perform the AR special effect. Control the AR device to display the effect.

Illustratively, for different application scenarios, the target physical object may be an object with a specific shape, such as a physical object such as a book, a painting, or a building. In a different application, the target physical object may be a physical object in the application. An AR special effect can provide an introduction to the physical object and improve the user's understanding of the physical object. At least one of an AR special effect having a relationship and an AR special effect having a relationship relationship with the content of the target entity object.

Exemplarily, an AR special effect having a predetermined relative positional relationship with a target physical object can represent that the AR special effect and the target physical object have a predetermined relative positional relationship in the same coordinate system. In some embodiments of the present application, AR special effects may include 3D AR screens and audio content. Here, the 3D AR screen may be generated in a pre-constructed 3D scene model. In the 3D scene model, data such as the shape, size, position, and attitude of the 3D AR screen, the positional relationship and relative attitude relationship between the AR screen and the target entity object, and the 3D data corresponding to the 3D scene model. A predetermined transformation relationship between the coordinate system and the world coordinate system in which the target physical object is located may be preset. Thus, after determining the position data of the target physical object in the world coordinate system, the position data of the AR screen in the world coordinate system can be determined based on the position data of the target physical object in the world coordinate system. Also, it is possible to simultaneously determine the posture data of the AR screen in the world coordinate system based on the posture data of the target entity object in the world coordinate system.

In some embodiments of the present application, the positional relationship and attitude relationship between the target physical object and the three-dimensional screen of the AR special effect may be further preset, such as the same coordinate system for the AR screen and the target physical object. A positional relationship and a pose relationship in (which may be a pre-established world coordinate system) may be set. Thus, after obtaining the position data of the target entity object in the world coordinate system, the position data of the AR screen in the world coordinate system can be determined, and after obtaining the pose data of the target entity object in the world coordinate system, It is also possible to determine the attitude data of the AR screen in the world coordinate system (angles formed between each designated direction in the AR screen and the X-, Y-, and Z-axes of the world coordinate system). In the process, there is no need to build a 3D scene model, which is more convenient and faster in the display process for AR special effects.

For example, when the target entity object is a calendar, the center of the calendar is the origin, the long side passing through the center of the calendar is the X axis, the short side passing through the center of the calendar is the Y axis, and the center of the calendar is A world coordinate system may be established with the Z-axis being a straight line passing through and perpendicular to the calendar cover. If the positional relationship between the AR screen and the target entity object includes that the AR screen is displayed on the upper surface of the calendar at a location where the distance from the center point of the calendar is a predetermined distance, the calendar in the world coordinate system After determining the position data, the display position of the AR screen can be determined based on this. The orientation data of the AR screen in the world coordinate system can also be determined based on the orientation data of the calendar in the world coordinate system and the preset orientation relationship between the AR screen and the calendar.

Exemplarily, the AR special effect having a related relationship with the content of the target entity object is an AR screen in which the display content of the AR special effect attracts the user's understanding of the content of the target entity object, the role that the target entity object has, or the target entity object. may refer to including In some embodiments of the present application, when the target entity object is a calendar, the display content of the AR special effect is an AR screen related to the year corresponding to the calendar, and audio content introducing the content contained in the calendar. and so on.

Illustratively, if the current scene image is identified to include a target entity object, then in a first positioning scheme, determining position information of the target entity object in the current scene image; The first display position data of the AR special effect can be determined based on the position information of the . Here, the first positioning method is a positioning method using a marker, that is, the image of the target entity object is used as the marker to determine the image position information of the target entity object in the current scene image. It is to be. In some embodiments, the first display pose data of the AR special effect can also be determined based on the position information of the target physical object in the current scene image. Based on the image identification technology, the position information of the target physical object in the current scene image can be determined relatively accurately. Therefore, here, based on the position information of the target physical object, the first display position data and the first display attitude data of the AR special effect can be obtained relatively accurately, so that the accurate display of the AR special effect can be obtained. provide support for

Illustratively, the first display pose data of the AR special effects can also be determined. If it is identified that the current scene image includes the target physical object, determine the first display position data and the first display orientation data of the AR special effect matching the target physical object by a first positioning scheme, and The AR device is controlled to display the AR special effect based on the display position data and the first display orientation data.

In S103, in the process of displaying the AR special effect, in response to the target physical object not being identified in the current scene image, determining second display position data of the AR special effect, and based on the second display position data; to control the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect.

In the process of controlling the AR device to display the AR special effect, movement of at least one of the target entity object and the AR device may cause at least one of the following two cases. That is, in Case 1, the relative position data between the target physical object and the AR device changes. In Case 2, the relative pose data between the target physical object and the AR device changes. In this case, the AR device may not be able to capture the target entity object or the complete target entity object when capturing the current scene image. Thus, when identifying the current scene image, there may be cases where the target physical object cannot be identified.

In some embodiments of the present application, the cases that the target physical object cannot be identified may include two cases, one case is that the current scene image does not contain the target physical object, another case is , the current scene image includes a part of the target entity object, for example, if the target entity object is a calendar, the current scene image includes only one corner of the calendar. In this case, the target physical object cannot be identified because it may not be possible to detect enough feature points contained in the target physical object.

If the current scene image includes a target physical object, determining the target first display position data by the first positioning method is performed based on the position information of the target physical object in the current scene image to perform the first display of the AR special effect. Considering that the position data is to be determined, in the process of positioning the target physical object according to the first positioning method, the relative position data between the AR device and the target physical object when taking each scene image can be simultaneously determined and the relative position data saved. In this way, when the target entity object is not identified in the current scene image, the stored relative position data of the AR device and the target entity object are combined with Simultaneous Localization And Mapping (SLAM) technology. Meanwhile, the relative position data between the AR device and the target physical object when taking the current scene image can be determined. In some embodiments of the present application, second display position data of the AR special effect can be determined based on the relative position data and the relative positional relationship between the AR special effect and the target physical object. The process will be detailed later.

Also, in the process of displaying the AR special effect, in response to the target physical object not being identified in the current scene image, a second positioning scheme may be used to determine second display pose data for the AR special effect. Here, the second positioning method is the SLAM positioning method. Then, based on the second display position data and the second display orientation data, the AR device is controlled so as to continue displaying the AR special effect according to the displayed progress of the AR special effect. Here, the determination process of the second display orientation data is similar to the determination process of the second display position data.

Exemplarily, the target physical object is a calendar, and the AR special effect includes a dynamically displayed AR screen with a total duration of 30s. When the AR screen is displayed until the 10th s, if the target physical object is not identified in the current scene image taken by the AR device, then the second display of the AR special effect determined by the second positioning method Based on the position data, or based on the second display position data and the second display orientation data, the AR device can be controlled to continue displaying according to the displayed progress of the AR screen (continue displaying from the 10th second). . In the process of continuing to display, the second display position data indicates that the AR device is completely away from the display position range of the AR screen, for example, the relative distance between the AR device and the calendar is greater than or equal to a predetermined threshold; Or, if the second display attitude data indicates that the shooting angle of the AR device is completely away from the calendar, in this case, the AR special effects continue to be displayed, but the user can change the AR special effects by the AR device. I can't appreciate the AR screen. In the process of continuing to display, it is determined, based on the second display position data, that the relative distance between the AR device and the calendar is less than a predetermined threshold, and that the shooting angle of the AR device can capture a part of the calendar. In this case, the user can view a part of the AR screen displayed by the AR device, for example, an AR screen that matches a part of the calendar.

In an embodiment of the present application, the AR device is controlled to display the AR special effect based on the determined first display position data when the target physical object is identified in the current scene image. In the process of displaying the AR special effect to the user, if the target physical object is not identified in the current scene image, determine second display position data of the AR special effect, and display the AR special effect based on the second display position data. AR equipment can also be controlled to continue displaying AR special effects that have not been displayed, which can improve the consistency and stability of AR special effects in the display process and make the display of AR special effects more realistic. .

FIG. 2 shows a schematic diagram of a system architecture in which the display method in an augmented reality scene of an embodiment of the present application can be applied. As shown in FIG. 2, the system architecture includes a current scene image acquisition terminal 201 , a network 202 and a control terminal 203 . To realize support for one exemplary application, the current scene image acquisition terminal 201 and the control terminal 203 establish a communication connection over the network 202, and the current scene image acquisition terminal 201 communicates with the network 202. The control terminal 203 reports the current scene image to the control terminal 203 via the control terminal 203, and if the control terminal 203 identifies that the target entity object is included in the current scene image, the control terminal 203 generates the first display position data of the AR special effect that matches the target entity object. and controlling the AR device to display the AR special effect based on the first display position data, and determining that the target entity object is not identified in the current scene image in the process of displaying the AR special effect In response, determining second display position data of the AR special effect, and controlling the AR device to continue displaying the AR special effect according to the displayed progress of the AR special effect based on the second display position data. Finally, the control terminal 203 uploads the display position information and the AR special effect to the network 202 and transmits them to the current scene image acquisition terminal 201 via the network 202 .

By way of example, the current scene image acquisition terminal 201 may include an image collection device, and the control terminal 203 may include a vision processing device or remote server with vision information processing capabilities. Network 202 may use wired or wireless connection schemes. Here, if the control terminal 203 is a vision processing device, the current scene image acquisition terminal 201 may communicate with the vision processing device by a wired connection method, and perform data communication via a bus, for example. If the control terminal 203 is a remote server, the current scene image acquisition terminal 201 can perform data interaction with the remote server through a wireless network.

Or, in some scenarios, the current scene image acquisition terminal 201 may be a vision processing device with a video acquisition module, or a host with a camera. In this case, the display method in the augmented reality scene of the embodiments of the present application may be performed by the current scene image acquisition terminal 201, and the above system architecture may not include the network 202 and the control terminal 203.

In some embodiments of the present application, when it is identified that the current scene image includes a target entity object, the following method is performed before determining the first display position data of the AR special effect that matches the target entity object. can identify whether the current scene image contains the target entity object. As shown in FIG. 3, the following S201-S202 are included.

In S201, feature point extraction is performed on the current scene image to obtain feature information corresponding to each of a plurality of feature points included in the current scene image. To position.

Illustratively, in the process of performing identification on a current scene image, first, an image detection algorithm can locate a target detection region containing a physical object in the current scene image. Subsequently, in the target detection area, feature point extraction is performed, and for example, feature points located on the outline of the entity object, feature points located in the sign pattern area, feature points located in the character area, etc. are extracted in the target detection area. You may Illustratively, the feature points may be uniformly extracted based on the position region corresponding to the target physical object in the current scene image, so that the extracted feature points can completely represent the target physical object. For example, if the target entity object is a calendar, it may be uniformly extracted in the rectangular area corresponding to the calendar cover in the current scene image.

Exemplarily, the feature information included in the feature points extracted here includes information capable of representing features of the feature points, such as texture feature values, RGB feature values, and grayscale values corresponding to the feature points. It's okay.

In step S202, comparison is performed based on the feature information corresponding to each of the plurality of feature points and the feature information corresponding to each of the plurality of feature points included in the target entity object stored in advance, and the target entity object is added to the current scene image. determines whether the is included.

Exemplarily, in the same manner, the target physical object may be photographed in advance, and feature information corresponding to each of a plurality of feature points included in the target physical object may be obtained and stored.

Exemplarily, when the comparison is performed based on the feature information corresponding to each of the plurality of feature points and the feature information corresponding to each of the plurality of feature points included in the pre-stored target entity object, first, the current scene image determining a first feature vector corresponding to the target detection region in the current scene image based on the feature information corresponding to each of the plurality of feature points extracted in step 1, and determining a first feature vector corresponding to each of the plurality of feature points included in the target entity object A second feature vector corresponding to the target physical object may be determined based on the corresponding feature information. Subsequently, the similarity between the target detection region and the target entity object may be determined by the first feature vector and the second feature vector, for example, by cosine equation.

Exemplarily, if it is determined that the similarity between the first feature vector and the second feature vector is greater than or equal to a predetermined similarity threshold, it is determined that the current scene image includes the target entity object. Conversely, if it is determined that the similarity between the first feature vector and the second feature vector is less than the predetermined similarity threshold, then it is determined that the current scene image does not contain the target entity object.

In an embodiment of the present application, a plurality of feature points included in the target detection area are extracted to identify whether the current scene image includes the target entity object, and a feature point comparison method is used to determine whether the target entity object is included in the current scene image. It can quickly and accurately determine whether an entity object is included.

Regarding S102 above, determining the first display position data of the AR special effect that matches the target physical object may include the following S301-S303, as shown in FIG.

In S301, the position information of the target entity object in the current scene image is obtained.

Exemplarily, an image coordinate system is established according to the current scene image, image coordinate values of a plurality of feature points included in the target entity object in the image coordinate system are obtained, and position information of the target entity object in the current scene image is obtained. be able to.

In S302, determine the position data of the target physical object in a pre-established world coordinate system according to the position information, and determine the position data of the AR device in the world coordinate system according to the current scene image.

In some embodiments of the present application, based on the position information, the transformation relationship between the image coordinate system and the camera coordinate system corresponding to the AR device, and the transformation relationship between the camera coordinate system corresponding to the AR device and the world coordinate system, Position data of the target physical object in the world coordinate system can be determined.

Illustratively, the camera coordinate system corresponding to the AR device may be a three-dimensional Cartesian coordinate system established with the origin at the focal center of the image acquisition member included in the AR device and the optical axis as the Z-axis. After the AR device captures the current scene image, the position data of the target physical object in the camera coordinate system can be determined according to the transformation relationship between the image coordinate system and the camera coordinate system.

Illustratively, a pre-established world coordinate system may be established with the center point of the target physical object as the origin. For example, when the target substantive object is a calendar as mentioned above, the center of the calendar is the origin, the long side passing through the center of the calendar is the X axis, the short side passing through the center of the calendar is the Y axis, A straight line passing through the center of the calendar and perpendicular to the calendar cover may be established as the Z-axis.

Here, the conversion between the camera coordinate system and the world coordinate system is a rigid body conversion, that is, a conversion method that allows the camera coordinate system to overlap with the world coordinate system by rotating and translating. A transformation relationship between the camera coordinate system and the world coordinate system may be determined by position coordinates in the world coordinate system and corresponding position coordinates in the camera coordinate system of a plurality of position points in the target physical object. Here, after obtaining the position information of the target entity object in the camera coordinate system, the position data of the target entity object in the world coordinate system is determined based on the transformation relationship between the camera coordinate system corresponding to the AR device and the world coordinate system. can do.

Illustratively, the position data of the AR device in the world coordinate system may be determined by the current scene image taken by the AR device. For example, select a feature point in the current scene image, the position coordinates of the selected feature point in the world coordinate system established by the target entity object, and the position of the selected feature point in the camera coordinate system corresponding to the AR device. By determining the coordinates, it is possible to determine the position data of the AR device in the world coordinate system when the current scene image is taken.

In S303, first display position data is determined based on the position data of the target entity object in the world coordinate system and the position data of the AR device in the world coordinate system.

Considering that the AR special effect and the target physical object have a predetermined positional relationship in the same coordinate system, here, based on the position data of the target physical object and the AR device in the same coordinate system, an AR special effect for the AR device First display position data for the effect can be determined.

Illustratively, movement of at least one of the target physical object and the AR device causes a change in the first display position data, eg, a change in the position data of the target physical object in the world coordinate system. AR devices can display AR special effects that change as the target physical object's position data changes. Alternatively, after the position data of the AR device in the world coordinate system is changed, the AR device can display an AR special effect that changes with the change of the position data of the AR device. After the simultaneous movement of the target physical object and the AR device causes a change in the relative position data, it also causes a change in the first display position data of the AR special effect, thereby changing the display of the AR special effect. In such a manner, a more realistic AR experience can be provided to the user. For example, when the orientation of the AR device moves from the left side of the target physical object to the right side of the target physical object, the user can see that the AR device has caused a corresponding transformation of the display of the AR special effect.

In some embodiments of the present application, the process of determining first presentation pose data is similar to determining first presentation position data.

In the embodiments of the present application, by determining the position data of the target entity object and the AR device in the same world coordinate system, the first display position data of the AR special effect for the AR device in the same world coordinate system is more accurately determined. , thus making the augmented reality scene displayed by the AR device more realistic.

In an embodiment of the present application, for S303, determining the first display position data based on the position data of the target entity object in the world coordinate system and the position data of the AR device in the world coordinate system is as shown in FIG. , may include the following S3031-S3032.

At S3031, the position data of the AR special effect in the world coordinate system is determined based on the position data of the target entity object in the world coordinate system.

Exemplarily, according to the position data of the target entity object in the world coordinate system and the preset positional relationship between the AR special effect and the target entity object in the same coordinate system (see the above description for details), the world coordinates Positional data for AR special effects in the system can be determined.

In S3032, first display position data is determined based on the position data of the AR special effect in the world coordinate system and the position data of the AR device in the world coordinate system.

Illustratively, if the AR special effect includes an AR screen, the position data of the AR special effect in the world coordinate system may include the position of the AR screen in the world coordinate system. Here, the position of the AR screen in the world coordinate system may be represented by the coordinate value of the center point of the AR screen in the world coordinate system. In this case, the determined orientation of the AR screen in the world coordinate system may be represented by the angle between the designated direction of the AR screen and each coordinate axis of the world coordinate system.

In some embodiments of the present application, the position data of the AR device in the world coordinate system may include the position of the image acquisition member in the AR device in the world coordinate system. Here, the position of the image acquisition member in the world coordinate system may be represented by coordinate values of a predetermined position point of the image acquisition member in the world coordinate system.

In the embodiment of the present application, when determining the above-mentioned first display orientation data, the orientation of the image acquisition member in the world coordinate system is the angle formed by the direction of the camera in the image acquisition member and each coordinate axis of the world coordinate system. may be represented.

Illustratively, the first display position data may be determined by the position of the AR special effect in the world coordinate system and the position of the AR device in the world coordinate system. The first display pose data may be determined by the pose of the AR special effect in the world coordinate system and the pose of the AR device in the world coordinate system. Then, based on the current scene image, the position data of the target physical object and the AR device in the world coordinate system can be accurately determined, thereby obtaining the first display position data of the AR special effect accurately and quickly. be able to.

In S103 above, determining the second display position data of the AR special effect may include the following S401-S402, as shown in FIG.

In S401, based on the current scene image, the past scene image, and the relative position data of the AR device and the target entity object in the pre-established world coordinate system when taking the past scene image, the AR when taking the current scene image Determine relative position data between the instrument and the target physical object.

For example, hereinafter, taking the current scene image as an example of the third scene image captured by the AR device, when capturing the current scene image based on the SLAM technology, the AR device and the target entity object A brief description of how the relative position data is determined is provided.

Select from the world coordinate system established with the center point of the target entity as the origin and the first frame of the scene image captured by the AR device from the time the AR device captures the scene image including the target entity object in the first frame. Positional information of the AR device in the world coordinate system when the scene image of the first frame is captured is determined based on the positional coordinates of each feature point in the world coordinate system and the camera coordinate system corresponding to the AR device. It can further include the position information of the target physical object in the world coordinate system when the AR device takes the scene image of the first frame, which is determined at the same time. Based on the position information of the AR device in the world coordinate system when the scene image of the first frame is taken, and the position information of the target entity object in the world coordinate system when the AR device takes the scene image of the first frame, The relative position data between the AR device and the target physical object in the pre-established world coordinate system when taking the first frame of the scene image can be determined.

In some embodiments of the present application, when the AR device captures the second frame of the scene image, the target feature points included in the first frame of the scene image are found in the second frame of the scene image, and the AR device captures these two frames. Based on the positional information of the target feature points in the camera coordinate system when each scene image is taken, the position offset amount of the AR device when taking the second frame scene image with respect to the first frame scene image is taken can decide. Subsequently, based on the position offset amount and the relative position data between the AR device and the target entity object in the world coordinate system established in advance when the scene image of the first frame was captured, the scene image of the second frame is generated. Determine the relative position data between the AR device and the target physical object in a pre-established world coordinate system at the time of shooting.

In some embodiments of the present application, the same method can be used to determine the positional offset amount of the AR device when capturing the current scene image relative to capturing the second frame scene image. In this way, the positional offset amount of the AR device when capturing the current scene image with respect to when capturing the second frame scene image, and the positional offset amount of the AR device in the pre-established world coordinate system when capturing the second frame scene image. The relative position data with the target physical object can be combined to determine the relative position data between the AR device and the target physical object in a pre-established world coordinate system when the current scene image is taken.

Also, based on the relative orientation data between the AR device and the target entity object in the pre-established world coordinate system when capturing the current scene image, the past scene image, and the past scene image, the AR when capturing the current scene image Relative pose data between the device and the target physical object can also be determined. Here, the relative attitude data determination process is similar to the relative position data determination process described above.

At S402, second display position data for the AR special effect is determined based on the relative position data.

Exemplarily, considering that the AR special effect and the target physical object have a predetermined positional relationship in the same coordinate system, here is also the relative Second display position data of the AR special effects for the AR device can be determined based on the target position data.

In the embodiments of the present application, using the current scene image, the past scene image, and the relative position data of the AR device and the target entity object in the world coordinate system when taking the past scene image, the AR when taking the current scene image The relative position data between the device and the target physical object can be determined relatively accurately, and thus the second display position data of the AR special effect can be determined based on the accurate relative position data. It can be realized that the AR special effects can be displayed even if the target physical object has not been identified.

In some embodiments of the present application, the AR special effect may further include an AR screen and audio content matching the AR screen, and for S103 above, display the AR special effect based on the second display position data. Controlling the AR device to continue displaying the AR special effect according to the completed progress is
If the target entity object is not identified in the current scene image and the display of the AR screen is not completed, the AR screen that is not displayed is displayed according to the displayed progress of the AR screen based on the second display position data. It may also include controlling the AR device to continue displaying matching audio content.

As an example, when the AR device leaves the display position range of the AR screen, if the display progress of the AR screen at this time has not ended, the progress of the AR screen at this time can continue. The AR screen cannot be viewed on some devices, and only the audio content that matches the AR screen is heard. This still gives the user an AR special effects experience. When the AR device returns to the display position range of the AR screen again, if the display of the AR screen is still not completed, it will display the undisplayed AR screen and the audio content matching the AR screen to the user. It can go on and give users a consistent AR experience.

In an embodiment of the present application, if the target entity object is not identified in the current scene image, and the AR screen display is not completed, the audio content matching the AR screen may continue to be displayed, and the AR special effect display is performed. , and can display AR special effects more realistically.

In some embodiments of the present application, in the process of displaying the AR special effect, in response to the target physical object not being identified in the current scene image, determining second display position data for the AR special effect; 2 After controlling the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect based on the display position data, the method according to the embodiments of the present application includes:
If the target entity object is re-identified in the current scene image, control the AR device to continue displaying the AR special effect according to the displayed progress of the AR special effect, again based on the determined first display position data. further includes

Illustratively, in the process of controlling the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect based on the second display position data, for the current scene image taken by the AR device: , can continue to perform image identification to identify whether the current scene image contains the target physical object. If the target entity object is re-identified in the current scene image, still continue to display the AR special effect according to the displayed progress of the AR special effect according to the first display position data determined by the first positioning method. Control the AR device to continue.

In the embodiment of the present application, after the target entity object is re-identified, the AR special effect is displayed again based on the first display position data with relatively high accuracy, so that the AR special effect in the display process is It can improve consistency and stability, and make the display of AR special effects more realistic.

In some embodiments of the present application, before controlling the AR device to display the AR special effect:
Obtaining an AR special effect that matches the target physical object, wherein the AR special effect includes special effect data corresponding to each of the plurality of virtual objects.

Illustratively, when a target physical object is identified for the first time, an AR special effect matching the target physical object can be obtained.

Special effect data corresponding to each virtual object in an AR special effect may include data such as the shape, color, and corresponding audio content of the virtual object when displayed by the AR device.

When controlling AR equipment to display AR special effects,
It includes controlling the AR device to sequentially display the special effect data corresponding to each of the plurality of virtual objects according to the display order of the special effect data corresponding to each of the plurality of virtual objects.

By way of example, if the AR special effect includes multiple virtual objects, the display order of each virtual object when displayed on the AR device may be set in advance. Display data of special effect data corresponding to each of a plurality of virtual objects may be determined based on attribute information of each virtual object and a preset display order of different attributes. Here, the attribute information may include static objects, dynamic objects, dynamic people, and the like.

In embodiments of the present application, an AR special effect composed of multiple virtual objects can be displayed to the user, making the display of the AR special effect more vivid.

In some embodiments of the present application, the target physical object includes a calendar, and before controlling the AR device to display the AR special effect, the display method according to the embodiments of the present application includes:
Further comprising obtaining an AR special effect that matches the calendar, the AR special effect including first special effect data generated based on the cover content of the calendar.

The first special effect data here may be first special effect data corresponding to each of a plurality of virtual objects corresponding to the cover content of the calendar.

Illustratively, the AR special effects include virtual objects such as a cartoon character dragon, a cartoon character squirrel, a virtual calendar cover that wraps the calendar cover, virtual characters, auspicious clouds, etc., and each virtual object when displayed by AR The display order may further include.

When controlling AR equipment to display AR special effects,
If the calendar cover content is identified, controlling the AR device to display an AR special effect matching the calendar cover content based on the first special effect data.

Exemplarily, when the calendar cover content is identified, first trigger and display a virtual calendar cover wrapping the calendar cover based on the first special effect data and the set display order, and then move on. The virtual title of the calendar is displayed in the form of a square, followed by the appearance of Zuiun, and then the cartoon character dragon and cartoon character squirrel are displayed on the virtual calendar cover, and the calendar is introduced in detail in the form of dialogue, completing the introduction. After that, the display of the AR special effect ends.

FIG. 7 shows an AR screen displayed in the process of controlling the AR device to display AR special effects that match the calendar cover content. The user can comprehend the calendar by viewing the AR screen and listening to the corresponding audio content.

In an embodiment of the present application, when the calendar cover is identified, the calendar cover can display an AR special effect to introduce the calendar, enrich the display content of the calendar, and improve the user's viewing interest in the calendar.

In an embodiment of the present application, the target entity object includes a calendar, and before controlling the AR device to display the AR special effect, the display method according to an embodiment of the present application includes:
Obtaining an AR special effect that matches the calendar, wherein the AR special effect includes second special effect data generated based on mark events at the same time in the past of at least one predetermined date in the calendar. Including further.

Illustratively, the calendar contains a number of predetermined dates on which a particular event occurred at the same time in the past. For example, January 1st is New Year's Day, and AR special effects can be generated based on events that occurred on New Year's Day in the past. The second special effect data in the AR special effect may include virtual characters, audio contents, virtual screens, etc. generated based on events occurring at the same time in the past. may further include the display order of

When controlling AR equipment to display AR special effects,
If at least one predetermined date in the calendar is identified, controlling the AR device to display an AR special effect matching the at least one predetermined date in the calendar based on the second special effect data. It's okay.

Exemplarily, when the current scene image captured by the AR device includes a predetermined date, the AR may display and introduce events that occurred at the same time in the past on the predetermined date according to the second special effect data. You can control the equipment.

In the embodiment of the present application, when a calendar is displayed to the user and a predetermined date in the calendar is acquired, the user is displayed with an AR special effect corresponding to the predetermined date to enrich the display content of the calendar. can also be

In the embodiment of the above method, the order of describing each step means a strict execution order and does not limit the implementation process in any way, and the specific execution order of each step is determined by its function and possible inherent logic. This should be understood by those skilled in the art.

According to the same technical idea, the embodiments of the present application further provide a display device in augmented reality scene corresponding to the display method in augmented reality scene. The principle for solving the problems by the apparatus in the embodiments of the present application is similar to the above display method in the embodiments of the present application, so the implementation of the apparatus can refer to the implementation of the method.

Referring to FIG. 8, FIG. 8 is a structural schematic diagram of a display device 800 in an augmented reality scene according to an embodiment of the present application. The display device
an acquisition module 801 configured to acquire a current scene image taken by an AR device;
determining first display position data of an AR special effect that matches the target physical object if the current scene image includes the target physical object, and displaying the AR special effect based on the first display position data; a first control module 802 configured to control the AR device to
In the process of displaying the AR special effect, in response to the target physical object not being identified in the current scene image, determining second display position data for the AR special effect, and based on the second display position data, displaying the AR a second control module 803 configured to control the AR device to keep displaying the AR special effect according to the displayed progress of the special effect.

In some embodiments of the present application, the acquisition module 801 performs feature point extraction on the current scene image, obtains feature information corresponding to each of a plurality of feature points included in the current scene image, and obtains a plurality of feature points. The point is located in the target detection area in the current scene image, based on the feature information corresponding to each of the plurality of feature points and the feature information corresponding to each of the plurality of feature points included in the pre-stored target entity object. It is configured to identify whether the current scene image includes the target entity object in the manner of making the comparison and determining whether the current scene image includes the target entity object.

In some embodiments of the present application, the first control module 802 obtains the position information of the target physical object in the current scene image, and based on the position information, calculates the position data of the target physical object in a pre-established world coordinate system. and determining the position data of the AR device in the world coordinate system based on the current scene image, and based on the position data of the target entity object in the world coordinate system and the position data of the AR device in the world coordinate system, the first configured to determine display position data;

In some embodiments of the present application, the first control module 802 determines the position information, the transformation relationship between the image coordinate system and the camera coordinate system corresponding to the AR device, and the camera coordinate system corresponding to the AR device and the world coordinate system. and the first control module 802 determines position data of the target physical object in the world coordinate system based on the transformation relationship of It is configured to determine position data of the AR special effect and to determine first display position data based on the position data of the AR special effect in the world coordinate system and the position data of the AR device in the world coordinate system.

In some embodiments of the present application, the second control module 803 determines the relative positions of the AR device and the target entity object in a pre-established world coordinate system when taking the current scene image, the past scene image, and the past scene image. Based on the data, determine relative position data between the AR device and the target physical object when taking the current scene image, and determine second display position data of the AR special effect based on the relative position data. Configured.

In some embodiments of the present application, the AR special effect includes an AR screen and audio content matching the AR screen, the second control module 803 determines that no target physical object is identified in the current scene image, and If the display of the AR screen has not been completed, the AR device is controlled to continue displaying the audio content matching the AR screen that is not displayed according to the displayed progress of the AR screen based on the second display position data. configured as

In some embodiments of the present application, the first control module 802, when the target physical object is re-identified in the current scene image, again based on the first display position data determined by the first positioning scheme, performs the AR It is configured to control the AR device to keep displaying the AR special effect according to the displayed progress of the special effect.

In some embodiments of the present application, before the first control module 802 is configured to control the AR device to display the AR special effects, the acquisition module 801 generates AR special effects that match the target physical object. configured to obtain an effect, the AR special effect includes special effect data corresponding to each of the plurality of virtual objects, the first control module 802 displaying the special effect data corresponding to each of the plurality of virtual objects; The AR device is configured to sequentially display the special effect data corresponding to each of the plurality of virtual objects according to the order.

In some embodiments of the present application, the target physical object includes a calendar, and before the first control module 802 is configured to control the AR device to display the AR special effect, the acquisition module 801: configured to obtain an AR special effect that matches the calendar, the AR special effect includes first special effect data generated based on the cover content of the calendar; is identified, the AR device is configured to display an AR special effect matching the cover content of the calendar based on the first special effect data.

In some embodiments of the present application, the target physical object includes a calendar, and before the first control module 802 is configured to control the AR device to display the AR special effect, the acquisition module 801: configured to obtain an AR special effect that matches the calendar, the AR special effect including second special effect data generated based on mark events at the same time in the past on at least one predetermined date in the calendar. and the first control module 802 is configured to display an AR special effect matching the at least one predetermined date in the calendar based on the second special effect data if the at least one predetermined date in the calendar is identified. to control the AR device.

The description of the processing flow of each module and the interaction flow between each module in the apparatus may refer to the related descriptions in the above method embodiments, and the detailed description is omitted here.

Corresponding to the display method in the augmented reality scene in FIG. 1, the embodiments of the present application further provide an electronic device 900. FIG. Referring to FIG. 9, FIG. 9 is a structural schematic diagram of an electronic device 900 according to an embodiment of the present application. The electronic device
A processor 91, a memory 92, and a bus 93, wherein machine readable instructions executable by the processor 91 are stored in the memory 92, and when the electronic device 900 operates, the processor 91 and the memory 92 are connected to the bus. 93 and executes the display method in the augmented reality scene of any one of the above embodiments when the machine readable instructions are executed by the processor 91. a memory 92 configured to store execution instructions; It includes an internal memory 921 and an external memory 922, where the internal memory 921 is also referred to as an internal memory, so as to temporarily store operation data in the processor 91 and data exchanged with the external memory 922 such as a hard disk. configured, processor 91 exchanges data with external memory 922 via internal memory 921, and when electronic device 900 operates, processor 91 and memory 92 communicate via bus 93 to processor 91. ,
Obtaining a current scene image captured by an augmented reality AR device, and determining the first display position data of the AR special effect matching the target entity object if the current scene image is identified to include the target entity object. , controlling the AR device to display the AR special effect based on the first display position data, in response to the target physical object not being identified in the current scene image in the process of displaying the AR special effect; , determine second display position data of the AR special effect, and cause execution of instructions to control the AR device to continue to display the AR special effect according to the displayed progress of the AR special effect based on the second display position data. .

Embodiments of the present application further provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium and, when executed by a processor, performs the method of displaying in an augmented reality scene as described in the above method embodiments. Among them, the storage medium may be a volatile or non-volatile computer-readable storage medium.

Embodiments of the present application further provide computer programs. The computer program includes computer readable code, and when the computer readable code is executed in the electronic device, the processor of the electronic device performs the method of displaying in an augmented reality scene as described in any one of the embodiments above.

Embodiments of the present application further provide a computer program product. The computer program product may include program code, instructions included in the program code may be configured to perform the method of displaying in an augmented reality scene as described in the method embodiments above, specifically Reference may be made to the method examples.

Among them, the computer program product may be realized by hardware, software or a combination thereof. In some embodiments, the computer program product may be embodied as a computer storage medium, and in some other embodiments, the computer program product is, for example, a Software Development Kit (SDK). It may be embodied as a software product such as.

Apparatus according to embodiments of the present application may be at least one of a system, a method and a computer program product. A computer program product may comprise a computer readable storage medium having computer readable program instructions stored thereon for causing a processor to implement aspects of the present application.

A computer-readable storage medium may be a tangible device capable of holding or storing instructions for use in an instruction-executing device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination of the above. Examples of computer readable storage media (non-exhaustive list) include portable computer disks, hard disks, Random Access Memory (RAM), Read-Only Memory (ROM), erasable programmable read-only Memory (Electrical Programmable Read-Only Memory (EPROM) or Flash, Static Random-Access Memory (SRAM), Portable Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc ( Digital Video Disc (DVD), memory sticks, floppy disks, punch cards on which instructions are stored or mechanical encoding devices such as protrusion structures in recesses, and any suitable combination of the above. Computer-readable storage media, as used herein, include radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or through electrical wires. It should not be construed as being a transitory signal per se, such as a transmitted electrical signal.

The computer readable program instructions described herein can be downloaded from a computer readable storage medium to each computing/processing device or can be downloaded from at least one of the Internet, local area networks, wide area networks and/or wireless networks. It can be downloaded to an external computer or external storage device via a network such as one. The network may include at least one of copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and edge servers. A network interface card or network interface at each computing/processing device receives computer-readable program instructions from the network, transfers the computer-readable program instructions for storage on a computer-readable storage medium at each computing/processing device.

Computer program instructions for performing the operations herein may be assembler instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or one or more It may be source code or target code written in any combination of programming languages. The programming languages include object-oriented programming languages such as Smalltalk, C++, etc., and traditional procedural programming languages such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user computer, partially executed on the user computer, executed as a separate software package, or partially executed on the user computer. It may be executed locally and partially executed on a remote computer, or completely executed on a remote computer or server. When referring to a remote computer, the remote computer connects to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or It can be connected to an external computer (eg, through the Internet using an Internet service provider). In some embodiments, state information in computer readable program instructions is used to customize electronic circuits, such as programmable logic circuits, FPGAs, or Programmable Logic Arrays (PLAs). The electronic circuitry may implement aspects of the present application by executing computer readable program instructions.

It is clearly understood by those skilled in the art that the specific working steps of the above-described systems and devices can refer to the corresponding steps in the method embodiments for convenience and simplification of description. should. It should be understood that the systems, devices and methods disclosed in some of the embodiments provided by the present invention can be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, for example, the division of the units is merely the division of logic functions, and other division schemes may be used in actual implementation. Also, for example, multiple units or components may be combined or incorporated into another system. Or some features may be ignored or not implemented. Also, the mutual couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some communication interface, device or unit, whether electrical, mechanical or Other forms are also possible.

The units described as separate members may or may not be physically separate. Members shown as units may or may not be physical units. That is, they may be located at the same location or distributed over a plurality of network units. Some or all of them can be selected according to actual needs to achieve the purpose of the measures in this embodiment.

Also, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist as a separate physical unit, or two or more units may be may be integrated in one unit.

The functionality may be implemented in the form of software functional units and stored in a non-volatile computer-readable storage medium executable by a processor when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present application is essentially or the part that contributed to the prior art or part of the technical solution is embodied in the form of a software product. The computer software product may be stored in a single storage medium, and may be stored in a single computer device (such as a personal computer, server, or network device) to perform the methods described in each embodiment of the present application. contains some instructions for executing all or part of the steps of The storage media mentioned above include various media capable of storing program codes, such as USB memory, removable hard disk, ROM, RAM, magnetic disk, or optical disk.

Finally, I would like to mention that the above examples are only specific embodiments of the present application, and are for the purpose of describing the technical solutions of the present application, not limiting them, and The scope of protection is not limited to this. Although the present application has been described in detail with reference to the above embodiments, any person skilled in the art who is familiar with the technical field can understand the technical solutions described in the above embodiments within the technical scope disclosed in the present application. Modifications can be made to or changes can be easily conceived, or some technical features can be replaced by equivalents, and these modifications, changes or replacements are It should be understood that the essence of the technical solution to be used is not departed from the spirit and scope of the technical solution in the embodiments of the present application, but shall fall within the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Embodiments of the present application provide a display method, apparatus, apparatus, medium and program in an augmented reality scene. The method is performed by an electronic device, and the display method includes obtaining a current scene image taken by an AR device; determining first display position data of an AR special effect that matches a physical object, and controlling the AR device to display the AR special effect based on the first display position data; and the AR special effect. in the process of displaying, in response to the target physical object not being identified in the current scene image, determining second display position data for the AR special effect; based on the second display position data; controlling the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect.

Claims (14)

A display method in an augmented reality scene performed by an electronic device, comprising:
obtaining a current scene image taken by an augmented reality (AR) device;
determining first display position data of an AR special effect that matches the target physical object if the current scene image includes a target physical object; and based on the first display position data, the AR controlling the AR device to display a special effect;
In the process of displaying the AR special effect, determining second display position data of the AR special effect in response to the target physical object not being identified in the current scene image; controlling the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect based on. If the current scene image is identified to include a target physical object, before determining first display position data for an AR special effect that matches the target physical object;
performing feature point extraction on the current scene image to obtain feature information corresponding to each of a plurality of feature points included in the current scene image, wherein the plurality of feature points is a target detection region in the current scene image located in
A comparison is performed based on feature information corresponding to each of the plurality of feature points and feature information corresponding to each of a plurality of feature points included in the target entity object stored in advance, and the current scene image is compared with the target entity object. In a manner that determines whether an object is included,
2. The method of claim 1, further comprising identifying whether the current scene image includes the target entity object. Determining first display position data for an AR special effect that matches the target physical object includes:
obtaining location information of the target entity object in the current scene image;
determining position data of the target entity object in a pre-established world coordinate system based on the position information, and determining position data of the AR device in the world coordinate system based on the current scene image; ,
determining the first display position data based on position data of the target physical object in the world coordinate system and position data of the AR device in the world coordinate system. Or the method of 2. Determining position data of the target physical object in a pre-established world coordinate system based on the position information includes:
The target in the world coordinate system based on the position information, the transformation relationship between the image coordinate system and the camera coordinate system corresponding to the AR device, and the transformation relationship between the camera coordinate system corresponding to the AR device and the world coordinate system determining location data of the entity object;
Determining the first display position data based on position data of the target entity object in the world coordinate system and position data of the AR device in the world coordinate system includes:
determining position data of the AR special effect in the world coordinate system based on position data of the target physical object in the world coordinate system;
and determining the first display position data based on position data of the AR special effect in the world coordinate system and position data of the AR device in the world coordinate system. The method described in . Determining the second display position data of the AR special effect includes:
When taking a current scene image, based on the current scene image, the past scene image, and the relative position data of the AR device and the target entity object in a pre-established world coordinate system when taking the past scene image. determining relative position data between the AR device and the target physical object;
determining second display position data for the AR special effect based on the relative position data. The AR special effect includes an AR screen and audio content matching the AR screen, and based on the second display position data, the AR special effect continues to be displayed according to the displayed progress of the AR special effect. Controlling an AR device is
If the target entity object is not identified in the current scene image and the display of the AR screen is not completed, it is displayed according to the displayed progress of the AR screen based on the second display position data. 6. The method of any one of claims 1-5, comprising controlling the AR device to continue displaying audio content matching the AR screen that is not displayed. In the process of displaying the AR special effect, determining second display position data of the AR special effect in response to the target physical object not being identified in the current scene image; after controlling the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect based on
When the target entity object is re-identified in the current scene image, the AR special effect continues to be displayed according to the displayed progress of the AR special effect based on the determined first display position data. A method according to any preceding claim, further comprising controlling the AR device. Before controlling the AR device to display the AR special effect, the method includes:
obtaining an AR special effect that matches the target physical object, wherein the AR special effect includes special effect data corresponding to each of a plurality of virtual objects;
Controlling the AR device to display the AR special effect includes:
and controlling the AR device to sequentially display the special effect data corresponding to each of the plurality of virtual objects according to the display order of the special effect data corresponding to each of the plurality of virtual objects. A method according to any one of claims 1-7. The target physical object includes a calendar, and before controlling the AR device to display the AR special effect, the method includes:
further comprising obtaining an AR special effect that matches the calendar, wherein the AR special effect includes first special effect data generated based on cover content of the calendar;
Controlling the AR device to display the AR special effect includes:
and controlling the AR device to display the AR special effect matching the calendar cover content based on the first special effect data when the calendar cover content is identified. The method according to any one of claims 1 to 8. The target physical object includes a calendar, and before controlling the AR device to display the AR special effect, the method includes:
obtaining an AR special effect that matches the calendar, wherein the AR special effect is a second special effect generated based on mark events at the same time in the past on at least one predetermined date in the calendar. further comprising comprising the data;
Controlling the AR device to display the AR special effect includes:
If at least one predetermined date in the calendar is identified, the AR device is configured to display the AR special effect matching at least one predetermined date in the calendar based on the second special effect data. A method according to any one of claims 1 to 9, characterized in that it comprises controlling. A display device in an augmented reality scene,
an acquisition module configured to acquire a current scene image taken by an augmented reality (AR) device;
determining first display position data of an AR special effect that matches the target physical object if the current scene image includes a target physical object; and based on the first display position data, the AR a first control module configured to control the AR device to display a special effect;
In the process of displaying the AR special effect, determining second display position data of the AR special effect in response to the target physical object not being identified in the current scene image; a second control module configured to control the AR device to keep displaying the AR special effect according to the displayed progress of the AR special effect based on. An electronic device, the electronic device comprising a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device operates, the The processor and the memory communicate via a bus and perform the display method in an augmented reality scene according to any one of claims 1 to 10 when the machine readable instructions are executed by the processor. ,Electronics. A computer readable storage medium, on which a computer program is stored, said computer program being stored in a processor, the display method in an augmented reality scene according to any one of claims 1 to 10. computer readable storage medium for executing A computer program, said computer program causing a processor of an electronic device to execute the display method in an augmented reality scene according to any one of claims 1 to 10.

Images