WO2023124691A1 - 增强现实场景的展示 - Google Patents

增强现实场景的展示 Download PDF

Info

Publication number
WO2023124691A1
WO2023124691A1 PCT/CN2022/134689 CN2022134689W WO2023124691A1 WO 2023124691 A1 WO2023124691 A1 WO 2023124691A1 CN 2022134689 W CN2022134689 W CN 2022134689W WO 2023124691 A1 WO2023124691 A1 WO 2023124691A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual object
display
image
coordinate system
pose data
Prior art date
Application number
PCT/CN2022/134689
Other languages
English (en)
French (fr)
Inventor
李斌
欧华富
李颖楠
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Publication of WO2023124691A1 publication Critical patent/WO2023124691A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes

Definitions

  • the present disclosure relates to the technical field of augmented reality, in particular to a display of augmented reality scenes.
  • Augmented Reality (AR) technology superimposes a virtual model on the real world, thereby presenting the real environment and virtual objects in the same picture or space in real time.
  • AR Augmented Reality
  • Embodiments of the present disclosure provide a display method, device, AR device, and storage medium of an augmented reality scene.
  • an embodiment of the present disclosure provides a method for displaying an augmented reality scene, the method including: in response to identifying an identification object from the first image of the current scene, according to the virtual object corresponding to the identification object in the world The first pose data in the coordinate system, determine the second pose data of the virtual object in the screen coordinate system of the AR device; based on the second pose data, display the virtual object on the AR device; During the presentation of the virtual object, in response to the fact that the identification object is not recognized from the second image of the current scene, based on the preset target pose data in the screen coordinate system, in the AR The virtual object is displayed on the screen of the device.
  • an embodiment of the present disclosure provides an augmented reality scene display device, the device comprising: a first pose determination module configured to respond to identifying an identification object from the first image of the current scene, according to The first pose data of the virtual object corresponding to the identification object in the world coordinate system determines the second pose data of the virtual object in the screen coordinate system of the AR device; the first presentation module is configured to The second pose data is for displaying the virtual object on the AR device; the second display module is configured to, during the display process of the virtual object, in response to no recognition of the virtual object from the second image of the current scene
  • the identification object displays the virtual object on the screen of the AR device based on the preset target pose data in the screen coordinate system.
  • an embodiment of the present disclosure provides an AR device, including: a processor; and a memory, communicably connected to the processor, where the memory stores computer instructions that can be read by the processor.
  • the computer instructions are used to cause the processor to execute the method according to any implementation manner of the first aspect.
  • an embodiment of the present disclosure provides a storage medium storing computer instructions for causing a computer to execute the method according to any embodiment of the first aspect.
  • the target pose data in the coordinate system displays the virtual object on the screen of the AR device, so that the display effect of the virtual object will not be interrupted, improving the AR display effect and user viewing experience.
  • Fig. 1 is a schematic structural diagram of an AR device according to some implementations of the present disclosure.
  • Fig. 2a and Fig. 2b are schematic structural diagrams of an AR device according to some embodiments of the present disclosure.
  • Fig. 3 is a structural block diagram of an AR device according to some implementations of the present disclosure.
  • FIG. 4 is a flowchart illustrating a method in some embodiments according to the present disclosure.
  • Figure 5 is a schematic diagram illustrating a method in some embodiments according to the present disclosure.
  • FIG. 6 is a flowchart illustrating a method in some embodiments according to the present disclosure.
  • Figure 7 is a flow chart illustrating a method in some embodiments according to the present disclosure.
  • Figure 8 is a schematic diagram illustrating a method in some embodiments according to the present disclosure.
  • Figure 9 is a schematic diagram illustrating a method in some embodiments according to the present disclosure.
  • Figure 10 is a schematic diagram illustrating a method in some embodiments according to the present disclosure.
  • Figure 11 is a flow chart illustrating a method in some embodiments according to the present disclosure.
  • Figure 12 is a flowchart illustrating a method in some embodiments according to the present disclosure.
  • Fig. 13 is a structural block diagram of a display device according to some embodiments of the present disclosure.
  • a prefabricated marker object When displaying a virtual object in an augmented reality scene, a prefabricated marker object (Marker) can be used to determine the display manner of the virtual object.
  • the identification object is, for example, a preset graphic drawn on the surface of the article based on Hamming Code technology, and the preset graphic can be, for example, a cartoon image, a two-dimensional code, a bar code, and the like.
  • the AR device recognizes that the Marker appears in the current scene image through the camera, it can display the corresponding virtual object on the Marker surface or in the scene space where it is located.
  • the display effect of the virtual object will also be interrupted, so that the virtual object cannot be displayed. That is, only when the user captures the Marker with the AR device, can he/she watch the display effect of the virtual object.
  • the display effect of the virtual object will be interrupted, and the user expects to watch the complete display effect without interruption, resulting in poor display effect and user experience.
  • Embodiments of the present disclosure provide a display method, device, AR device, and storage medium of an augmented reality scene, aiming at improving the display effect of virtual objects in the AR scene and improving user experience.
  • the present disclosure provides a method for displaying an augmented reality scene, and the method can be applied to an AR device.
  • the AR device described in the embodiments of the present disclosure can be any terminal device with AR functions, such as head-mounted devices such as AR glasses and AR helmets, or handheld devices such as smart phones and tablet computers, or smart phones. Wrist-worn devices such as watches and smart bracelets, etc., are not limited in this disclosure.
  • the AR device described in the present disclosure is described by taking AR glasses as an example.
  • the AR device 100 includes a main body 101, and the main body 101 includes a bracket structure for wearing and supporting.
  • the display screen 110 is arranged on the main body 101 , and after the user wears the AR device 100 , the display screen 110 can be located directly in front of the user's eyes, so that the user can easily observe the images displayed on the display screen 110 .
  • the AR device 100 also includes a camera 120, which is used to collect images of the current scene.
  • the AR device 100 also includes a processor disposed inside the main body 101 , and the processor is used to superimpose the current scene image captured by the camera 120 with the virtual object, and display it on the display screen 110 .
  • the AR device described in the present disclosure is described by taking a smart phone as an example.
  • the AR device 100 includes a housing 102, which is a shell structure for supporting the main body of the mobile phone.
  • the display screen 110 is arranged on the front of the casing 102, and the camera 120 is arranged on the front and/or the back of the casing 102.
  • the camera 120 is arranged on the back of the casing.
  • the camera 120 is used to collect the current scene image, so that the processor of the smart phone superimposes the current scene image collected by the camera 120 with the virtual object, and displays it on the display screen 110 .
  • FIG. 3 shows a structural block diagram of an AR device in the present disclosure.
  • the AR device in an embodiment of the present disclosure will be further described below in conjunction with FIG. 3 .
  • the AR device 100 includes a processor 130 , a memory 140 , a camera 120 , a display screen 110 and an IMU (Inertial Measurement Unit, inertial measurement unit) sensor 160 .
  • IMU Inertial Measurement Unit, inertial measurement unit
  • the processor 130 , the memory 140 , the camera 120 , the display screen 110 and the IMU sensor 160 establish a communicable connection between any of them through the bus 150 .
  • the processor 130 can be any type of processor with one or more processing cores. It can perform single-threaded or multi-threaded operations, and is used to parse instructions to perform operations such as obtaining data, performing logical operation functions, and delivering operation processing results.
  • the memory 140 may include a non-volatile computer-readable storage medium, such as at least one magnetic disk storage device, a flash memory device, a distributed storage device located remotely with respect to the processor 130, or other non-volatile solid-state storage devices.
  • the memory may have a program storage area for storing non-volatile software programs, non-volatile computer-executable programs and modules, which are invoked by the processor 130 to make the processor 130 execute one or more method steps below.
  • the memory 140 may also include a storage part such as a volatile random storage medium or a hard disk, as a data storage area for storing calculation processing results and data output by the processor 130 .
  • the camera 120 is used to collect the current scene image, so that the processor 130 can perform fusion processing for the virtual object according to the collected current scene image, so as to realize the AR display effect.
  • the camera 120 can be any type of camera suitable for implementation, such as a black and white camera, an RGB camera, etc.
  • the camera 120 can be divided into a monocular camera and a binocular camera according to the number of cameras, which is not limited in the present disclosure. .
  • the display screen 110 is configured to receive the display information issued by the processor 130 , so as to present a corresponding display image on the display screen 110 according to the display information, so that the user can observe the AR display effect through the display screen 110 .
  • the display screen 110 may be any display screen type suitable for implementation, such as an LCD (Liquid Crystal Display, liquid crystal display) display screen, an OLED (Organic Light-Emitting Diode, organic light-emitting semiconductor) display screen, etc., This disclosure does not limit this.
  • the IMU sensor 160 is mainly used to detect and measure the acceleration and rotation of the AR device itself. Its basic principle is to use the law of inertia to measure the acceleration and rotation. It can calculate the pose of the AR device itself in real time based on the sensor signal.
  • the IMU sensor 160 may include, for example, a three-axis or six-axis gyroscope, an acceleration sensor, and the like.
  • the display method of the augmented reality scene in the example of the present disclosure includes:
  • the image of the current scene refers to the scene image collected in real time by using the camera of the AR device.
  • the camera 120 collects scene images within the viewfinder range, and displays the collected scene images on the display screen 110 through data processing.
  • the image that the user watches in real time on the display screen 110 is An image of the current scene. It can be understood that as the pose of the AR device changes, the collected image of the current scene changes accordingly, that is, the image of the current scene is a real-time image.
  • the identified image containing the identified object may be referred to as the first image.
  • the identification object refers to the marker used to trigger the display of the AR effect, that is, the above-mentioned Marker.
  • the specific presentation mode of the identification object may be, for example, a QR code, a preset graphic, etc., which is not limited in the present disclosure.
  • the identification object 210 is a cartoon graphic drawn on the surface of the postcard 200.
  • the cartoon graphic can be drawn based on Hamming code technology, so that when the AR device captures the cartoon graphic, it can identify the cartoon graphic. Graphics for identification and analysis.
  • the process of drawing and analyzing Hamming codes can be understood and fully implemented by those skilled in the art by referring to related technologies, which is not limited in the present disclosure.
  • the identification object is a preset graphic drawn on the surface of the object.
  • the AR device captures the scene image including the preset graphic, it can extract and identify the feature points of the scene image through image detection technology, so that The preset image is recognized from the scene image.
  • image detection technology so that The preset image is recognized from the scene image.
  • the identification object is associated with the corresponding virtual object.
  • a model database including at least one virtual object can be established in advance, and the virtual object in the model database is associated with the corresponding identification object, so that the AR device can be viewed from the current scene.
  • the virtual object corresponding to the identification object can be determined by searching the model database based on the corresponding relationship.
  • a virtual object is represented as a virtual image rendered and presented on the display screen of an AR device.
  • the display effect of the interaction between the virtual object and the real scene can be presented.
  • the AR device detects and recognizes the captured image of the current scene through image detection technology, and when the identified object is identified from the image of the current scene, the virtual object corresponding to the identified object can be determined.
  • the principle is to find the mapping relationship between the pose information of the virtual object in the established world coordinate system and the pose information in the screen coordinate system of the AR device.
  • the pose information of the virtual object in the world coordinate system is defined as the first pose data
  • the pose information of the virtual object in the screen coordinate system of the AR device is defined as the second pose data.
  • features of the marked object can be extracted in advance based on image detection technology to obtain at least one preset key point of the marked object, and a world coordinate system can be established based on the at least one preset key point.
  • the world coordinate system can be aligned with the screen coordinate system of the AR device (for example, through translation and/or rotation) to obtain a mapping relationship between the two coordinate systems.
  • the second pose data of the virtual object in the screen coordinate system can be determined through the mapping relationship, and realized based on the second pose data
  • the virtual object after determining the second pose data of the virtual object in the screen coordinate system of the AR device, the virtual object can be displayed at the corresponding position on the screen according to the second pose data, so that the user can The virtual object is observed at the corresponding location on the screen.
  • the virtual object has the above-mentioned mapping relationship between the world coordinate system and the screen coordinate system, as the identification object moves, the pose of the virtual object moves accordingly, and the position of the virtual object in the screen coordinate system corresponds to The movement occurs, so that the user can observe the visual effect that the virtual object is adsorbed on the identification object.
  • the virtual object viewed on the screen of the AR device moves along with the surface of the postcard 200 .
  • the corresponding virtual object can be viewed on the display screen of the AR device. Display of results.
  • the display effect of the virtual object is not directly interrupted at this time, but the virtual object is continued to be displayed based on the preset target pose data.
  • target pose data may be preset in the screen coordinate system of the AR device, and the target pose data represents data related to the display position and pose of the virtual object in the screen coordinate system. Therefore, when the AR device 100 cannot recognize the identified object in the second image, the display of the virtual object will not be interrupted, but the virtual object will continue to be displayed with the preset target pose data.
  • the target pose data includes placement data of the virtual object.
  • the virtual object may be displayed in the center of the display screen of the AR device based on the display position data.
  • the virtual object may be displayed on the lower right corner of the display screen of the AR device based on the display location data.
  • the display position data of the virtual object on the display screen of the AR device can be positioned in real time by using Simultaneous Localization And Mapping (SLAM) technology. This disclosure does not limit this.
  • SLAM Simultaneous Localization And Mapping
  • the target pose data includes display pose data of the virtual object.
  • the display pose data may indicate that the virtual object rotates at the display position; another example, the display pose data may indicate that the virtual object performs a preset action at the display position;
  • the display pose data may be the same as the pose data included in the second pose data, that is, the virtual object continues to be displayed in the previous pose; etc., the disclosure does not limit this.
  • the display position of the virtual object on the screen of the AR device can be determined based on the display position data, and the display posture of the virtual object can be determined based on the display posture data, so as to display the virtual object at the corresponding display position with the display posture .
  • the AR device may redisplay the AR effect based on the methods described in S410 and S420 above. This is described in the following embodiments of the present disclosure, and will not be described in detail here.
  • the display method of the embodiment of the present disclosure in the AR effect display scene, if the identification object cannot be recognized in the second image of the current scene, the display of the virtual object will not be interrupted, so that there is no need to check the position of the user using the AR device.
  • Restrictions can give users a higher degree of freedom.
  • the user when a user wears the AR glasses shown in FIG. 1 and visits a scene where identified objects are arranged, the user expects to watch the complete display content of the virtual object corresponding to a certain identified object. At this time, there is no need to limit the direction of the camera of the AR glasses to the position of the marked object. During the display of the virtual object, the user can move. Even if the AR glasses cannot capture the marked object, the complete image of the virtual object can still be displayed on the display. content, so that users can watch the complete display content and improve user experience.
  • the AR device when the AR effect is displayed, the AR device needs to recognize the identification object from the collected image of the current scene.
  • the process of identifying an object from an image of the current scene includes:
  • S610 Perform feature point extraction on the image of the current scene to obtain feature information corresponding to at least one feature point included in the image of the current scene.
  • the AR device 100 can collect an image of the current scene through the camera 120. It can be understood that the image of the current scene can be a single-frame image collected by the camera, or a multi-frame image in a video stream collected by the camera. This disclosure does not limit this.
  • the image detection algorithm can be used to locate the detection area including the object to be tested in the image of the current scene, and then extract one or more feature points of the object to be tested in the detection area .
  • the image detection algorithm can be used to first determine the area of the postcard 200 as the area to be tested, and then perform feature point extraction on the "Little Dragon Man” and "Auspicious Cloud” included on the postcard 200 to be tested. Obtain the feature information of each object to be tested.
  • the feature information of the object to be tested can be compared with the feature point information of the pre-stored identification object. If the similarity between the two is greater than the preset threshold, it means that the object to be tested is an identification object; otherwise, if the similarity is not greater than the preset threshold, it means that the object to be tested is not the identified object.
  • the feature point information described in the embodiments of the present disclosure may include texture feature value, RGB feature value, gray value and other information that can identify the feature point feature corresponding to the feature point, which is not limited in the present disclosure.
  • the virtual object corresponding to the identification object and the pose data required for displaying the virtual object can be further obtained.
  • the following will describe the embodiment in conjunction with FIG. 7 .
  • the process of determining the second pose data of the virtual object includes:
  • S720 Determine second pose data of the virtual object in the screen coordinate system according to the first pose data and the mapping relationship between the world coordinate system and the screen coordinate system of the AR device.
  • the identification object is a cartoon image drawn on the surface of the postcard 200
  • feature extraction can be performed on the marked object based on the image detection technology to obtain at least one preset key point of the marked object, and then a world coordinate system is established according to the extracted preset key point.
  • the preset key point may be the center point of the identified object, and the center point of the identified object is used as the origin O of the coordinate system, and the direction parallel to the short side of the postcard 200 and passing through the origin O is used as the
  • the X-axis takes the direction parallel to the long side of the postcard 200 and passes through the origin O as the Y-axis, and takes the direction perpendicular to the X-axis and Y-axis and passes through the origin O as the Z-axis, thereby establishing the world coordinate system O-XYZ.
  • the process of establishing the world coordinate system in FIG. 5 is only an example of the implementation of the present disclosure.
  • the world coordinate system can also be established according to other preset key points, for example, the vertex in the lower left corner of the postcard 200 can be used as
  • the coordinate origin O establishes a coordinate system, which is not limited in the present disclosure.
  • the world coordinate system can be aligned with the screen coordinate system of the AR device, so as to construct a mapping relationship between the world coordinate system and the screen coordinate system.
  • the world coordinate system can be mapped to the camera coordinate system of the AR device through rotation and translation, and then mapped from the camera coordinate system to the screen coordinate system of the AR device.
  • the preset display effect includes display effect data required to display the virtual object, such as model rendering data of the virtual object and pose data of the virtual object.
  • the preset display effect is: the cartoon image "little dragon man” rotates at the position shown in the illustration.
  • the display effect data includes model rendering data required for rendering the cartoon image and pose data representing the rotation of the cartoon image at a preset position.
  • the AR device can determine the first pose data of the virtual object in the world coordinate system according to the preset display effect data.
  • the display effect data indicates that the virtual object continues to rotate at the origin O of the world coordinate system. Therefore, the first pose data determined by the AR device based on the display effect data includes the position data and posture of the virtual object Data, where the position data is the origin O(0,0,0), and the attitude data is rotation-related data.
  • the AR device After determining the first pose data, the AR device can map the first pose data of the virtual object in the world coordinate system to the screen coordinate system according to the aforementioned mapping relationship, and obtain the first pose data of the virtual object in the screen coordinate system. Two pose data. After determining the second pose data, the AR device can display the virtual object on the display screen according to the second pose data.
  • the AR effect displayed on the AR device may refer to FIG. 8 , the virtual object presents a three-dimensional image on the surface of the postcard 200, and the virtual object presents a rotation effect at the corresponding position.
  • the display position of the virtual object "Little Dragon" on the display screen of the AR device is determined based on the mapping from the pose data in the world coordinate system to the screen coordinate system. Therefore, when the pose of the postcard 200 changes, the established world coordinate system changes accordingly, and the pose of the virtual object changes following the change of the world coordinate system.
  • the corresponding display effect is that the virtual object is adsorbed on the surface of the postcard 200, and its pose changes with The pose of the postcard 200 changes.
  • the user moves the postcard 200 shown in FIG. 5 out of the field of view of the camera of the AR device in the real world.
  • the image of the current scene collected by the AR device at a certain moment is shown in Figure 9.
  • the image of the current scene has not been recognized or only incomplete identification objects 210 have been recognized.
  • the virtual object can be displayed on the display screen of the AR device based on the method flow described in FIG. 4 and according to the preset target pose data.
  • FIG. 10 A presentation method according to an embodiment of the present disclosure is illustrated in FIG. 10 .
  • the virtual object is displayed in the center of the display interface of the display screen 110.
  • FIG. 10 is only used as an exemplary display effect.
  • those skilled in the art can also realize any other display effect suitable for implementation based on the inventive concepts disclosed in the present disclosure. Enumerate again.
  • the AR display effect is coherent
  • the coherence of the AR display content can be maintained. The following will describe the embodiment in conjunction with FIG. 11 .
  • the display method of the example of the present disclosure further includes:
  • the displayed content often has a display progress.
  • the content displayed by the AR device can be a video or animation introducing cultural relics.
  • the playback progress of the currently displayed video or animation can be marked, so that through the display method described in the above-mentioned implementation mode, at the preset position of the display screen of the AR device with The current playback progress continues to display the virtual object, that is, to realize the continuation of the playback, to ensure the continuity of the displayed content before and after the identification object disappears.
  • the display content may also include audio data, and the audio data may also be played continuously based on the above method, so that whether it is the screen content or audio content seen by the user, it is coherent before and after the identification object disappears. Improve AR display effect.
  • the virtual object is continuously displayed on the screen of the AR device to ensure the continuity of the displayed content before and after the identification object disappears, and improve the AR display effect and user experience.
  • the virtual object in the case that the identification object cannot be recognized in the image of the current scene, the virtual object can be displayed without interruption according to the process of the above embodiment.
  • the process of the above-mentioned embodiment may also be re-executed, so that the virtual object once again presents the display effect of being attracted to the identification object.
  • the demonstration method of the present disclosure example also includes:
  • the identification object or the complete identification object cannot be recognized in the image of the current scene, so that the virtual object is displayed at the preset position of the display screen 110 of the AR device.
  • the position of the AR device remains unchanged, and the postcard 200 gradually moves to the right, so that the identified object is re-recognized in the image of the current scene captured by the camera of the AR device.
  • the image in which the identified object is re-identified may be referred to as a third image, and in response to identifying the identified object in a third image acquired after the second image of the current scene, according to the Identifying third pose data of the virtual object corresponding to the object in the world coordinate system, and determining fourth pose data of the virtual object in the screen coordinate system of the AR device.
  • the steps of S410-S420 can be re-executed based on the process described in the embodiment of FIG. 4, so that the virtual object can be re-displayed on the surface of the postcard based on the second pose data, showing the display effect of being adsorbed on the surface of the postcard.
  • the specific process those skilled in the art can understand and fully implement it with reference to the foregoing implementation manners, and the present disclosure will not repeat it here.
  • the second pose data can be determined based on the identification object again, and the virtual object can be displayed to enhance the interaction effect and improve the display effect and user experience.
  • different display effects may be rendered for the virtual object for the case where the identified object is recognized and the case where the identified object is not recognized. The following will be described with reference to the embodiment shown in FIG. 12 .
  • the display method of the present disclosure example also includes:
  • the first display effect is different from the second display effect. Because the first display effect is different from the second display effect, it may be determined based on the display effect whether the identification object is recognized from the image of the current scene.
  • the image in which the target object is recognized is referred to as a first image
  • the image in which no identification object is recognized is referred to as a second image.
  • the display effect may include multiple dimensions, such as brightness, transparency, blurring degree, size, display position, display angle, etc., and the first display effect may be different from the second display effect in at least one dimension.
  • the display effect of the virtual object can be as shown in FIG.
  • the degree of transformation, position and size are displayed according to the aforementioned second pose data.
  • the virtual object is no longer displayed with the first display effect, but is displayed with the second display effect.
  • the transparency of the virtual object increases, changing from opaque to translucent; another example, the size of the virtual object becomes smaller, shrinking from the original size to half its size; another example, the brightness of the virtual object becomes darker, and the original brightness is reduced to half Brightness; etc., which are not enumerated in this disclosure.
  • the user wears the AR glasses shown in FIG. 1 to visit an exhibition hall where the identification objects are pre-arranged.
  • the AR glasses recognize the identification objects in the first image of the current scene, they can (First display effect) Display the virtual object.
  • the AR glasses do not recognize the identified object in the second image of the current scene, so the virtual object is displayed on the display screen of the AR glasses with 50% transparency (second display effect).
  • the display screen of the AR glasses displays the virtual object again in an opaque manner.
  • the embodiment of the present disclosure provides an augmented reality scene display device, which can be applied to an AR device.
  • the AR device may be, for example, the AR glasses shown in FIG. 1 .
  • the AR device may be, for example, a smart phone as shown in Fig. 2a and Fig. 2b. This disclosure does not limit this.
  • the display device of the augmented reality scene of the example of the present disclosure includes:
  • the first pose determination module 10 is configured to, in response to identifying the identified object from the first image of the current scene, determine the virtual object in the AR according to the first pose data of the virtual object corresponding to the identified object in the world coordinate system.
  • the first display module 20 is configured to display the virtual object on the AR device based on the second pose data
  • the second presentation module 30 is configured to respond to the fact that the identification object is not recognized from the second image of the current scene during the presentation of the virtual object, based on the preset target pose data in the screen coordinate system, in the AR A virtual object is displayed on the screen of the device.
  • the second display module 30 is configured to:
  • the second display module 30 is configured to:
  • the fourth pose data of the virtual object in the screen coordinate system of the AR device based on the fourth pose data, display the virtual object on the AR device.
  • the first display module 20 is configured to: display the virtual object with a first display effect when the identification object is recognized from the first image of the current scene;
  • the second display module 30 is configured to: in the case that the identification object is not recognized from the second image of the current scene, display the virtual object with a second display effect, the second display effect is different from the first display effect, so as to distinguish from A situation in which the identification object is recognized in the current scene image and a situation in which the identification object is not recognized in the current scene image.
  • the target pose data includes display position data and display pose data for the virtual object; the second display module 30 is configured to:
  • the virtual object is presented in the posture indicated by the presentation posture data.
  • the first pose determination module 10 is configured to:
  • the AR device determine the second pose data of the virtual object in the screen coordinate system.
  • the first pose determination module 10 is configured to:
  • the feature information corresponding to the at least one feature point is compared with the pre-stored feature point information of the identified object to identify the identified object in the first image of the current scene.
  • the display of the virtual object will not be interrupted, thereby improving the AR display effect and improving user experience.
  • Different virtual object display effects are set before and after the identification object disappears. On the one hand, it enhances interactivity and fun, and on the other hand, it reminds the user that the identification object disappears from the current scene image, improving user experience.
  • An embodiment of the present disclosure provides an AR device, including:
  • the memory is communicably connected to the processor, and the memory stores computer instructions that can be read by the processor, and the computer instructions are used to make the processor execute the method according to any implementation manner of the first aspect.
  • Embodiments of the present disclosure provide a storage medium storing computer instructions, which are used to make a computer execute the method according to any embodiment of the first aspect.
  • This disclosure relates to the field of augmented reality.
  • acquiring the image information of the target object in the real environment and then using various visual correlation algorithms to detect or identify the relevant features, states and attributes of the target object, and to obtain the information that matches the specific application.
  • AR effect combining virtual and reality.
  • the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places.
  • Vision-related algorithms may involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc.
  • Specific applications can not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlays and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual Interactive scenarios such as model display.
  • the relevant features, states and attributes of the target object can be detected or identified through the convolutional neural network.
  • the above-mentioned convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本公开涉及一种增强现实场景的展示方法，包括响应于从当前场景的第一图像中识别到标识对象，根据标识对象对应的虚拟对象在世界坐标系中的第一位姿数据，确定虚拟对象在AR设备的屏幕坐标系的第二位姿数据，并基于第二位姿数据，在AR设备展示虚拟对象，在虚拟对象的展示过程中，响应于从当前场景的第二图像中未识别到标识对象，基于预先设置的在屏幕坐标系中的目标位姿数据，在AR设备的屏幕上展示虚拟对象。

Description

增强现实场景的展示
相关申请的交叉引用
本申请要求在2021年12月31日提交至中国专利局、申请号为CN2021116730875的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及增强现实技术领域，具体涉及一种增强现实场景的展示。
背景技术
增强现实(Augmented Reality，AR)技术通过将虚拟模型叠加到真实世界中，从而将真实的环境和虚拟的对象实时地在同一个画面或者空间呈现。对于AR场景，如何丰富虚拟对象与真实场景的交互，优化展示效果是主要的研究方向之一。
发明内容
本公开实施方式提供了一种增强现实场景的展示方法、装置、AR设备以及存储介质。
第一方面，本公开实施方式提供了一种增强现实场景的展示方法，所述方法包括：响应于从当前场景的第一图像中识别到标识对象，根据所述标识对象对应的虚拟对象在世界坐标系中的第一位姿数据，确定所述虚拟对象在AR设备的屏幕坐标系的第二位姿数据；基于所述第二位姿数据，在所述AR设备展示所述虚拟对象；在所述虚拟对象的展示过程中，响应于从所述当前场景的第二图像中未识别到所述标识对象，基于预先设置的在所述屏幕坐标系中的目标位姿数据，在所述AR设备的屏幕上展示所述虚拟对象。
第二方面，本公开实施方式提供了一种增强现实场景的展示装置，所述装置包括：第一位姿确定模块，被配置为响应于从当前场景的第一图像中识别到标识对象，根据所述标识对象对应的虚拟对象在世界坐标系中的第一位姿数据，确定所述虚拟对象在AR设备的屏幕坐标系的第二位姿数据；第一展示模块，被配置为基于所述第二位姿数据，在所述AR设备展示所述虚拟对象；第二展示模块，被配置为在所述虚拟对象的展示过程中，响应于从所述当前场景的第二图像中未识别到所述标识对象，基于预先设置的在所述屏幕坐标系中的目标位姿数据，在所述AR设备的屏幕上展示所述虚拟对象。
第三方面，本公开实施方式提供了一种AR设备，包括：处理器；和存储器，与所述处理器可通信连接，所述存储器存储有能够被所述处理器读取的计算机指令，所述计算机指令用于使所述处理器执行根据第一方面任一实施方式所述的方法。
第四方面，本公开实施方式提供了一种存储介质，存储有计算机指令，所述计算机指令用于使计算机执行根据第一方面任一实施方式所述的方法。
本公开实施方式的展示方法，在虚拟对象的展示过程中，响应于从当前场景图像中未识别到标识对象，也即标识对象由当前场景图像中丢失的情况下，可基于预先设置的在屏幕坐标系中的目标位姿数据，在AR设备的屏幕上展示虚拟对象，从而虚拟对象的展示效果不会中断，提高AR展示效果和用户观看体验。
附图说明
为了更清楚地说明本公开具体实施方式或相关技术中的技术方案，下面将对具体实施方式或相关技术的描述中所需要使用的附图作简单地介绍，下面描述中的附图是本公开的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是根据本公开一些实施方式中AR设备的结构示意图。
图2a和图2b是根据本公开一些实施方式中AR设备的结构示意图。
图3是根据本公开一些实施方式中AR设备的结构框图。
图4是根据本公开一些实施方式中展示方法的流程图。
图5是根据本公开一些实施方式中展示方法的原理图。
图6是根据本公开一些实施方式中展示方法的流程图。
图7是根据本公开一些实施方式中展示方法的流程图。
图8是根据本公开一些实施方式中展示方法的原理图。
图9是根据本公开一些实施方式中展示方法的原理图。
图10是根据本公开一些实施方式中展示方法的原理图。
图11是根据本公开一些实施方式中展示方法的流程图。
图12是根据本公开一些实施方式中展示方法的流程图。
图13是根据本公开一些实施方式中展示装置的结构框图。
具体实施方式
下面将结合附图对本公开的技术方案进行清楚、完整地描述，所描述的实施方式是本公开一部分实施方式，而不是全部的实施方式。基于本公开中的实施方式，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施方式，都属于本公开保护的范围。此外，下面所描述的本公开不同实施方式中所涉及的技术特征在彼此不冲突的情况下可以相互结合。
增强现实场景中对虚拟对象展示时，可以利用预先制作好的标识对象(Marker)来确定虚拟对象的展示方式。其中，标识对象例如为基于汉明码(Hamming Code)技术绘制在物品表面的预设图形等，预设图形可以是例如卡通形象、二维码、条形码等。AR设备通过摄像头识别到当前场景图像中出现该Marker时，可在Marker表面或者所在场景空间中展示对应的虚拟对象。
但是相关技术中，在虚拟对象的展示过程中，一旦Marker从当前场景中消失(例如被移出当前场景)，虚拟对象的展示效果也会被中断，从而无法展示虚拟对象。也即，只有用户利用AR设备拍摄到Marker时，才能观看到虚拟对象的展示效果。而在用户处于移动状态下，当AR设备拍摄不到Marker时，虚拟对象的展示效果会被中断，用户又期望不中断地观看完整的展示效果，导致展示效果和用户体验不佳。
本公开实施方式提供了一种增强现实场景的展示方法、装置、AR设备以及存储介质，旨在提高AR场景中虚拟对象的展示效果，提高用户体验。
在一些实施方式中，本公开提供了一种增强现实场景的展示方法，该方法可以应用 于AR设备。可以理解，本公开实施方式所述的AR设备，可以是任何具有AR功能的终端设备，例如AR眼镜、AR头盔等头戴式设备，又例如智能手机、平板电脑等手持式设备，再例如智能手表、智能手环等腕戴式设备等等，本公开对此不作限制。
如图1所示，在一个示例中，本公开所述的AR设备以AR眼镜为例进行说明。AR设备100包括主体101，主体101包括实现佩戴及支撑的支架结构。显示屏110设置在主体101上，用户在佩戴AR设备100之后，显示屏110可以位于用户眼睛的正前方，从而便于用户观察到显示屏110上显示的画面。AR设备100还包括摄像头120，摄像头120用于采集当前场景图像。AR设备100还包括设于主体101内部的处理器，处理器用于将摄像头120采集的当前场景图像与虚拟对象进行叠加之后，在显示屏110上展示。
如图2a所示，在一个示例中，本公开所述的AR设备以智能手机为例进行说明。AR设备100包括壳体102，壳体102为实现手机主体支撑的外壳结构。显示屏110设置在壳体102正面，摄像头120设置在壳体102的正面和/或背面，例如图2b所示，摄像头120设于壳体的背面。摄像头120用于采集当前场景图像，从而智能手机的处理器将摄像头120采集的当前场景图像与虚拟对象进行叠加之后，在显示屏110上展示。
当然可以理解，AR设备的具体设备类型不局限于上述示例的AR眼镜和智能手机，本公开对此不再赘述。图3中示出了本公开AR设备的结构框图，下面结合图3对本公开实施方式的AR设备进一步说明。
如图3所示，在一些实施方式中，AR设备100包括处理器130、存储器140、摄像头120、显示屏110以及IMU(Inertial Measurement Unit，惯性测量单元)传感器160。
处理器130、存储器140、摄像头120、显示屏110以及IMU传感器160通过总线150建立任意两者之间的可通信连接。
处理器130可以为任何类型，具备一个或者多个处理核心的处理器。其可以执行单线程或者多线程的操作，用于解析指令以执行获取数据、执行逻辑运算功能以及下发运算处理结果等操作。
存储器140可包括非易失性计算机可读存储介质，例如至少一个磁盘存储器件、闪存器件、相对于处理器130远程设置的分布式存储设备或者其他非易失性固态存储器件。存储器可以具有程序存储区，用于存储非易失性软件程序、非易失性计算机可执行程序以及模块，供处理器130调用以使处理器130执行下文一个或者多个方法步骤。存储器140还可以包括易失性随机存储介质、或者硬盘等存储部分，作为数据存储区，用以存储处理器130下发输出的运算处理结果及数据。
摄像头120用于采集当前场景图像，从而处理器130可以根据采集到的当前场景图像进行针对虚拟对象的融合处理，实现AR展示效果。在本公开实施方式中，摄像头120可以是任何适于实施的摄像头类型，例如黑白摄像头、RGB摄像头等，摄像头120按照摄像头的数量可以分为单目摄像头和双目摄像头，本公开对此不作限制。
显示屏110用于接收处理器130下发的显示信息，从而根据显示信息在显示屏110上呈现出对应的显示图像，以使用户可以通过显示屏110观察到AR展示效果。在本公开实施方式中，显示屏110可以是任何适于实施的显示屏类型，例如LCD(Liquid Crystal Display，液晶显示)显示屏、OLED(Organic Light-Emitting Diode，有机发光半导体)显示屏等，本公开对此不作限制。
IMU传感器160主要用来检测和测量AR设备自身的加速度与旋转运动，其基本原理是利用惯性定律实现加速度与旋转运动的测量，可以基于传感器信号实时对AR设备 自身位姿进行解算。在一些实施方式中，IMU传感器160可以包括例如三轴或六轴陀螺仪、加速度传感器等。
基于上述AR设备结构，下面对本公开实施方式的增强现实场景的展示方法进行说明。
如图4所示，在一些实施方式中，本公开示例的增强现实场景的展示方法包括：
S410、响应于从当前场景的图像中识别到标识对象，根据标识对象对应的虚拟对象在世界坐标系中的第一位姿数据，确定虚拟对象在AR设备的屏幕坐标系的第二位姿数据。
具体而言，当前场景的图像是指利用AR设备的摄像头实时采集到的场景图像。例如图1所示的AR眼镜中，通过摄像头120采集取景范围内的场景图像，通过数据处理将采集到的场景图像显示在显示屏110上，用户在显示屏110上实时观看到的图像即为当前场景的图像。可以理解，随着AR设备的位姿发生改变，采集到的当前场景的图像随之改变，也即当前场景的图像为实时图像。对于当前场景，识别到含标识对象的图像可以被称为第一图像。
标识对象指用于触发AR效果展示的标识物，也即上文所述的Marker，标识对象的具体呈现方式可以是例如二维码、预设图形等，本公开对此不作限制。
例如一个示例中，如图5所示，标识对象210为绘制于明信片200表面的卡通图形，该卡通图形可以基于例如汉明码技术进行绘制，从而AR设备在采集到该卡通图形时，可以对卡通图形进行识别解析。对于汉明码绘制和解析的过程，本领域技术人员参照相关技术即可理解并充分实施，本公开对此不作限制。
例如另一个示例中，标识对象为绘制于物体表面的预设图形，AR设备在采集到包括该预设图形的场景图像时，可以通过图像检测技术对场景图像进行特征点提取和识别，从而可以从场景图像中识别到该预设图像。对于图像检测和预设图像的识别过程，本领域技术人员参照相关技术即可理解并充分实施，本公开对此不作限制。
可以理解，标识对象关联对应的虚拟对象，例如一个示例中，可以预先建立包括至少一个虚拟对象的模型数据库，模型数据库中的虚拟对象关联与之对应的标识对象，从而在AR设备由当前场景的图像中识别到标识对象时，可以基于对应关系通过查找模型数据库，确定与该标识对象对应的虚拟对象。
虚拟对象表示为在AR设备的显示屏上经渲染呈现的虚拟形象，通过将虚拟对象与显示屏上显示的真实场景图像进行融合展示，可以呈现出虚拟对象与真实场景产生互动的展示效果。
本公开实施方式中，AR设备通过图像检测技术对采集到的当前场景的图像进行检测识别，在从当前场景的图像中识别到标识对象的情况下，即可确定与标识对象对应的虚拟对象。
在基于Marker进行AR展示时，其原理是找到虚拟对象在建立的世界坐标系中的位姿信息与在AR设备的屏幕坐标系的位姿信息的映射关系。本公开实施方式中，定义虚拟对象在世界坐标系中的位姿信息为第一位姿数据，定义虚拟对象在AR设备的屏幕坐标系中的位姿信息为第二位姿数据。
在一些实施方式中，可以基于图像检测技术预先对标识对象进行特征提取，得到标识对象的至少一个预设关键点，基于至少一个预设关键点建立世界坐标系。在得到世界坐标系之后，可将世界坐标系与AR设备的屏幕坐标系进行对齐处理(例如通过平移和/或旋转)，得到两个坐标系之间的映射关系。在对虚拟对象进行展示时，可以根据虚拟 对象在世界坐标系中的第一位姿数据，通过该映射关系确定虚拟对象在屏幕坐标系中的第二位姿数据，基于第二位姿数据实现虚拟对象在AR设备屏幕上的展示。本公开下文实施方式中对此进行具体说明，在此暂不详述。
S420、基于第二位姿数据，在AR设备展示虚拟对象。
具体而言，在确定虚拟对象在AR设备的屏幕坐标系中的第二位姿数据之后，即可根据该第二位姿数据将虚拟对象在屏幕中对应的位置进行展示呈现，以使得用户可以在屏幕的对应位置观察到该虚拟对象。
另外，可以理解，由于虚拟对象在世界坐标系与屏幕坐标系之间具有上述的映射关系，随着标识对象的移动，虚拟对象的位姿随之发生移动，虚拟对象在屏幕坐标系的位置相应发生移动，使得用户观看到虚拟对象吸附在标识对象上的视觉效果。例如，在真实场景中，当图5所示的明信片200移动时，在AR设备屏幕上观看到的虚拟对象随着明信片200的表面移动。
S430、在虚拟对象的展示过程中，响应于从当前场景的图像中未识别到标识对象，基于预先设置的屏幕坐标系中的目标位姿数据，在AR设备的屏幕上展示虚拟对象。
在一个示例场景中，以上述图5所示的明信片200为例，当用户通过AR设备识别到明信片200上的标识对象210时，即可在AR设备的显示屏上观看到对应的虚拟对象的展示效果。
当明信片200与AR设备100的相对位置发生改变时，例如，用户手持明信片200移动脱离AR设备100的取景范围，也即AR设备100的摄像头120在当前场景的第二图像中无法识别到标识对象210时，此时并非直接中断虚拟对象的展示效果，而是基于预先设置的目标位姿数据继续展示虚拟对象。
具体而言，可以在AR设备的屏幕坐标系中预先设置目标位姿数据，目标位姿数据表示虚拟对象在屏幕坐标系中的展示位置和姿态相关的数据。从而，在AR设备100在第二图像中无法识别到标识对象时，不会中断虚拟对象的展示，而是以预先设置的目标位姿数据继续展示虚拟对象。
在一些实施方式中，目标位姿数据包括虚拟对象的展示位置数据。例如一个示例中，基于展示位置数据可以将虚拟对象展示在AR设备显示屏的中央。例如另一个示例中，基于展示位置数据可以将虚拟对象展示在AR设备显示屏的右下角。例如又一个示例中，可以基于虚拟对象的初始位置，利用同步定位与建图(Simultaneous Localization And Mapping，SLAM)技术，实时定位虚拟对象在AR设备显示屏上的展示位置数据。本公开对此不作限制。
在一些实施方式中，目标位姿数据包括虚拟对象的展示姿态数据，例如展示姿态数据可以表示虚拟对象在展示位置旋转；又例如展示姿态数据可以表示虚拟对象在展示位置做出预设动作；再例如，展示姿态数据可以与第二位姿数据中包括的姿态数据相同，也即虚拟对象继续以之前的姿态进行展示；等等，本公开对此不作限制。
在对虚拟对象进行展示时，可基于展示位置数据确定虚拟对象在AR设备屏幕上的展示位置，基于展示姿态数据确定虚拟对象的展示姿态，从而以该展示姿态在对应的展示位置处展示虚拟对象。
在一些实施方式中，在当前场景的图像中重新出现该标识对象时，AR设备可以重新基于前述S410和S420所述的方式进行AR效果的展示。本公开下述实施方式中对此进行说明，在此暂不详述。
本公开实施方式的展示方法，在AR效果展示场景中，在当前场景的第二图像中无 法识别到标识对象的情况下，虚拟对象的展示不会中断，从而无需对用户使用AR设备的位置进行限制，可以给予用户更高的自由度。
例如一个示例场景中，用户佩戴图1所示的AR眼镜参观布置有标识对象的场景时，用户期望观看某个标识对象所对应的虚拟对象的完整展示内容。此时，无需将AR眼镜的摄像头的方向限定在该标识对象位置，在虚拟对象展示过程中，用户可以移动，即使AR眼镜采集不到标识对象，依旧可以在显示屏上展示虚拟对象的完整的内容，使得用户可以观看到完整的展示内容，提高用户体验。
通过上述可知，本公开实施方式中，在AR展示场景中，在当前场景的第二图像中无法识别到标识对象的情况下，虚拟对象的展示不会中断，提高AR展示效果，提高用户体验。
本公开实施方式中，在AR效果展示时，AR设备需要从采集到的当前场景的图像中识别出标识对象。如图6所示，在一些实施方式中，从当前场景的图像中识别标识对象的过程包括：
S610、对当前场景的图像进行特征点提取，得到当前场景的图像包括的至少一个特征点对应的特征信息。
S620、将至少一个特征点对应的特征信息与预先存储的标识对象的特征点信息进行对比，识别当前场景的图像中的标识对象。
具体而言，AR设备100可以通过摄像头120采集到当前场景的图像，可以理解，当前场景的图像可以是摄像头采集到的单帧图像，也可以是摄像头采集到的视频流中的多帧图像，本公开对此不作限制。
在对当前场景的图像进行处理的过程中，可以通过图像检测算法，定位出当前场景的图像中包括待测对象的检测区域，然后在检测区域中提取得到待测对象的一个或多个特征点。
以当前场景的图像为图5为例，可以通过图像检测算法，首先确定明信片200区域为待测区域，然后对明信片200上包括的“小龙人”和“祥云”等待测对象进行特征点提取，得到每个待测对象的特征信息。
在得到各个待测对象的特征信息之后，可以将待测对象的特征信息与预先存储的标识对象的特征点信息进行对比，若两者相似度大于预设阈值，则表示待测对象即为标识对象；反之，若相似度不大于预设阈值，表示待测对象不是标识对象。
仍以图5为例，假设预先存储的标识对象的特征点信息对应“小龙人”卡通形象，从而将当前场景的图像中提取到的小龙人的特征信息与预先存储的标识对象的特征点信息进行对比，两者相似度大于预设阈值，从而确定当前场景的图像中识别到标识对象。
示例性地，本公开实施方式所述的特征点的信息可以包括特征点对应的纹理特征值、RGB特征值、灰度值等能够标识该特征点特征的信息，本公开对此不作限制。
在从当前场景的图像中识别到标识对象时，即可进一步获取标识对象所对应的虚拟对象，以及该虚拟对象展示所需的位姿数据等。下面结合图7实施方式进行说明。
如图7所示，在一些实施方式中，本公开示例的展示方法，确定虚拟对象的第二位姿数据的过程包括：
S710、根据虚拟对象在世界坐标系中的预设位姿，确定第一位姿数据。
S720、根据第一位姿数据，以及世界坐标系与AR设备的屏幕坐标系的映射关系，确定虚拟对象在屏幕坐标系的第二位姿数据。
具体而言，以当前场景的图像为图5为例，标识对象为绘制于明信片200表面的卡通形象，当AR设备通过前述图6实施方式，对当前场景的图像检测识别到该图像包含标识对象时，可以基于图像检测技术对标识对象进行特征提取，得到标识对象的至少一个预设关键点，然后根据提取到的预设关键点建立世界坐标系。
例如一个示例中，如图5所示，预设关键点可以是标识对象的中心点，将标识对象的中心点作为坐标系原点O，以平行于明信片200短边且穿过原点O的方向作为X轴，以平行于明信片200长边且穿过原点O的方向作为Y轴，以垂直于X轴和Y轴且穿过原点O的方向作为Z轴，从而建立世界坐标系O-XYZ。
可以理解，图5中建立世界坐标系的过程仅作为本公开实施方式的一个示例，在其他实施方式中，还可以依据其他预设关键点建立世界坐标系，例如可以明信片200左下角的顶点作为坐标原点O建立坐标系，本公开对此不作限制。
继续参照图5，在建立世界坐标系之后，可以将世界坐标系与AR设备的屏幕坐标系进行对齐处理，从而构建出世界坐标系与屏幕坐标系之间的映射关系。例如一些实施方式中，可以将世界坐标系通过旋转和平移处理对应到AR设备的相机坐标系，然后从相机坐标系映射到AR设备的屏幕坐标系中。
对于世界坐标系与屏幕坐标系的对齐处理过程，本领域技术人员参照相关技术可以理解并充分实施，本公开对此不再赘述。
基于前述可知，在进行AR效果展示时，预先设置的展示效果中包括有展示虚拟对象所需的展示效果数据，例如虚拟对象的模型渲染数据以及虚拟对象的位姿数据等。例如图5示例中，预先设置的展示效果为：卡通形象“小龙人”在图示位置处旋转。从而展示效果数据中包括渲染卡通形象所需的模型渲染数据以及表示卡通形象在预设位置旋转的位姿数据。
从而，AR设备即可根据预先设置的展示效果数据确定虚拟对象在世界坐标系中的第一位姿数据。以虚拟对象“小龙人”为例，展示效果数据表示虚拟对象在世界坐标系的原点O位置持续旋转，从而，AR设备根据展示效果数据确定的第一位姿数据包括虚拟对象的位置数据和姿态数据，其中，位置数据为原点O(0,0,0)，姿态数据为旋转相关的数据。
在确定第一位姿数据之后，AR设备即可根据前述的映射关系，将虚拟对象在世界坐标系中的第一位姿数据映射到屏幕坐标系中，得到虚拟对象在屏幕坐标系中的第二位姿数据。在确定第二位姿数据之后，AR设备即可根据第二位姿数据在显示屏上展示虚拟对象。在图5示例中，AR设备上展示的AR效果可参照图8所示，虚拟对象在明信片200表面呈现立体形象，并且虚拟对象在对应的位置呈现旋转的效果。
以图8所示，虚拟对象“小龙人”在AR设备显示屏上的展示位置是基于世界坐标系中的位姿数据到屏幕坐标系中的映射确定的。从而，当明信片200发生位姿变化时，建立的世界坐标系随之变化，虚拟对象跟随世界坐标系变化产生位姿变化，对应的展示效果表现为虚拟对象吸附在明信片200表面，其位姿随明信片200的位姿变化而变化。
在一些示例场景中，用户在真实世界中将图5中所示的明信片200移出AR设备的摄像头的取景范围。在移动过程中，某个时刻AR设备采集的当前场景的图像如图9所示，此时，当前场景的图像中已经识别不到或者只能识别到不完整的标识对象210，在本公开实施方式中，即可基于前述图4所述的方法流程，根据预设的目标位姿数据，将虚拟对象展示在AR设备显示屏上。
图10中示出了根据本公开实施方式的展示方法。如图10所示，在本示例中，在当 前场景的图像中无法识别到标识对象的情况下，将虚拟对象在显示屏110的显示界面中央进行展示。
当然，可以理解，图10仅作为一种示例性的展示效果，在其他实施方式中，本领域技术人员基于本公开发明构思，还可以实现其他任何适于实施的展示效果，本公开对此不再枚举。
通过上述可知，本公开实施方式中，在AR展示场景中，在当前场景的图像中无法识别到标识对象的情况下，虚拟对象的展示不会中断，提高AR展示效果，提高用户体验。
在一些实施方式中，考虑到一些场景中，AR展示效果具有连贯性，为保证用户观看不中断，在当前场景的图像中无法识别到标识对象后，可以保持AR展示内容的连贯。下面结合图11实施方式进行说明。
如图11所示，在一些实施方式中，本公开示例的展示方法，还包括：
S1110、在虚拟对象的展示过程中，响应于从当前场景的图像中未识别到标识对象，确定当前已展示进度。
S1120、基于目标位姿数据和已展示进度，在AR设备的屏幕上继续展示虚拟对象。
具体而言，在虚拟对象的展示过程中，展示的内容往往具有展示进度。例如对于文物展览馆场景，用户扫描场景中预先布置的标识对象之后，AR设备展示的内容可以是一段文物介绍的视频或动画。
本本示例中，在当前场景的图像中无法识别到标识对象时，可以标记当前展示的视频或动画的播放进度，从而通过上述实施方式所述的展示方法，在AR设备的显示屏预设位置以当前播放进度继续展示虚拟对象，也即实现续播，保证标识对象消失前后展示内容的连贯性。
在一些实施方式中，展示内容中还可以包括音频数据，音频数据同样可以基于上述方式实现连续播放，从而，无论是用户看到的画面内容还是音频内容，在标识对象消失前后均是连贯的，提高AR展示效果。
通过上述可知，本公开实施方式中，根据目标位姿数据和已展示进度，在AR设备屏幕上继续展示虚拟对象，保证标识对象消失前后展示内容的连贯性，提高AR展示效果及用户体验。
在上述实施方式中，在当前场景的图像中识别不到标识对象的情况下，可以按照上述实施方式过程实现虚拟对象不中断展示。在一些实施方式中，在当前场景的第三图像中重新出现标识对象时，还可以重新执行上述实施方式的过程，使得虚拟对象再次呈现吸附在标识对象上的展示效果。
例如一些实施方式中，本公开示例的展示方法，还包括：
响应于从当前场景的图像中重新识别到标识对象，根据标识对象对应的虚拟对象当前在世界坐标系中的第一位姿数据，重新确定虚拟对象在AR设备的屏幕坐标系的第二位姿数据；基于第二位姿数据，重新在AR设备展示虚拟对象。
例如图10示例中，当前场景的图像中无法识别到标识对象或者无法识别到完整的标识对象，从而在AR设备的显示屏110的预设位置展示虚拟对象。在一个示例场景中，在真实场景中，AR设备位置保持不变，明信片200逐渐向右移动，从而AR设备的摄像头采集的当前场景的图像中重新识别到标识对象。在一些示例中，可以将重新识别到标识对象的图像称为第三图像，响应于从所述当前场景的所述第二图像之后采集的第三 图像中识别到所述标识对象，根据所述标识对象对应的所述虚拟对象在所述世界坐标系中的第三位姿数据，确定所述虚拟对象在所述AR设备的所述屏幕坐标系的第四位姿数据。
在此情况下，可以基于前述图4实施方式所述的过程，重新执行S410～S420的步骤，使得虚拟对象重新基于第二位姿数据在明信片表面展示，呈现为吸附在明信片表面的展示效果。对于具体过程，本领域技术人员参照前述实施方式即可理解并充分实施，本公开对此不再赘述。
通过上述可知，本公开实施方式中，在当前场景的图像重新识别到标识对象的情况下，可以重新基于标识对象确定第二位姿数据，并对虚拟对象进行展示，增强互动效果，提高展示效果和用户体验。
在一些实施方式中，为增强AR展示效果，可以针对识别到标识对象的情况和识别不到标识对象的情况，对虚拟对象渲染不同的展示效果。下面结合图12实施方式进行说明。
如图12所示，在一些实施方式中，本公开示例的展示方法还包括：
S1210、在从当前场景的图像中识别到标识对象的情况下，以第一展示效果展示虚拟对象。
S1220、在从当前场景的图像中未识别到标识对象的情况下，以第二展示效果展示虚拟对象。
具体而言，第一展示效果不同于第二展示效果。通过第一展示效果不同于第二展示效果，可以基于展示效果确定是否从当前场景的图像中识别到标识对象。在一些实施例中，将识别到目标对象的图像称为第一图像，将未识别到标识对象的图像称为第二图像。例如，对于文物展览馆场景，当在第一图像中识别到标识对象之后，用户可能会无意识地走动，导致AR设备采集的第二图像中未识别到标识对象，在这种情况下，通过第一展示效果不同于第二展示效果，可以提醒用户自身佩戴的AR设备已偏移。可以理解，展示效果可以包括多个维度，例如亮度、透明度、虚化程度、大小、展示位置、展示角度等等，第一展示效果与第二展示效果可以至少在一个维度上存在不同。
例如一个示例中，在当前场景的图像中可以识别到标识对象的情况下，虚拟对象的展示效果可如图8所示，也即虚拟对象的第一展示效果所包括的诸如亮度、透明度、虚化程度、位置以及大小均按照前述第二位姿数据进行展示。
在当前场景的图像中无法识别到标识对象的情况下，虚拟对象不再以第一展示效果进行展示，而是以第二展示效果进行展示。例如，虚拟对象的透明度增加，由不透明变换为半透明；又例如，虚拟对象的大小变小，由原始大小缩小为一半大小；再例如，虚拟对象的亮度变暗，有原始亮度变低为一半亮度；等等，本公开对此不再枚举。
在一个示例场景中，用户佩戴图1所示的AR眼镜参观预先布置有标识对象的展览馆，当AR眼镜在当前场景的第一图像中识别到标识对象，可根据前述实施方式以不透明的方式(第一展示效果)展示虚拟对象。随着用户位置移动，AR眼镜在当前场景的第二图像中未识别到该标识对象，从而在AR眼镜的显示屏上以50％透明度(第二展示效果)展示虚拟对象。当AR眼镜在当前场景的第二图像之后采集的第三图像中再次识别到该标识对象，AR眼镜的显示屏上再次以不透明的方式展示虚拟对象。
上述仅作为本公开实施方式的一种示例，在其他实施方式中，在标识对象消失前后，展示效果还可以具有其他区别，本公开对此不作限制。
通过上述可知，本公开实施方式中，针对标识对象消失前后设置不同的虚拟对象展 示效果，一方面增强互动性和趣味性，另一方面提示用户标识对象从当前场景的图像中消失，提高用户体验。
本公开实施方式提供了一种增强现实场景的展示装置，该装置可应用于AR设备。在一些实施方式中，AR设备可以是例如图1所示的AR眼镜。在另一些实施方式中，AR设备可以是例如图2a和图2b所示的智能手机。本公开对此不作限制。
如图13所示，在一些实施方式中，本公开示例的增强现实场景的展示装置，包括：
第一位姿确定模块10，被配置为响应于从当前场景的第一图像中识别到标识对象，根据标识对象对应的虚拟对象在世界坐标系中的第一位姿数据，确定虚拟对象在AR设备的屏幕坐标系的第二位姿数据；
第一展示模块20，被配置为基于第二位姿数据，在AR设备展示虚拟对象；
第二展示模块30，被配置为在虚拟对象的展示过程中，响应于从当前场景的第二图像中未识别到标识对象，基于预先设置的在屏幕坐标系中的目标位姿数据，在AR设备的屏幕上展示虚拟对象。
通过上述可知，本公开实施方式中，在AR展示场景中，在当前场景图像中无法识别到标识对象的情况下，虚拟对象的展示不会中断，提高AR展示效果，提高用户体验。
在一些实施方式中，第二展示模块30被配置为：
在虚拟对象的展示过程中，响应于从当前场景的第二图像中未识别到标识对象，确定当前已展示进度；
基于目标位姿数据和已展示进度，在AR设备的屏幕上继续展示虚拟对象。
在一些实施方式中，第二展示模块30被配置为：
响应于从当前场景的第二图像之后采集的第三图像中识别到标识对象，根据所述标识对象对应的所述虚拟对象当前在所述世界坐标系中的第三位姿数据，重新确定所述虚拟对象在所述AR设备的所述屏幕坐标系的第四位姿数据；基于第四位姿数据，在AR设备展示虚拟对象。
在一些实施方式中，第一展示模块20被配置为：在从当前场景的第一图像中识别到标识对象的情况下，以第一展示效果展示虚拟对象；
第二展示模块30被配置为：在从当前场景的第二图像中未识别到标识对象的情况下，以第二展示效果展示虚拟对象，第二展示效果不同于第一展示效果，以区别从所述当前场景图像中识别到所述标识对象的情况和从所述当前场景图像中未识别到所述标识对象的情况。
在一些实施方式中，目标位姿数据包括针对虚拟对象的展示位置数据和展示姿态数据；第二展示模块30被配置为：
在展示位置数据所指示的屏幕中的位置处，以展示姿态数据所指示的姿态展示虚拟对象。
在一些实施方式中，第一位姿确定模块10，被配置为：
根据虚拟对象在世界坐标系中的预设位姿，确定第一位姿数据；世界坐标系为根据标识对象的特征确定的；
根据第一位姿数据，以及世界坐标系与AR设备的屏幕坐标系的映射关系，确定虚拟对象在屏幕坐标系的第二位姿数据。
在一些实施方式中，第一位姿确定模块10，被配置为：
对当前场景的第一图像进行特征点提取，得到当前场景的第一图像包括的至少一个特征点对应的特征信息；
将至少一个特征点对应的特征信息与预先存储的标识对象的特征点信息进行对比，识别当前场景的第一图像中的标识对象。
通过上述可知，本公开实施方式中，在AR展示场景中，在当前场景图像中无法识别到标识对象的情况下，虚拟对象的展示不会中断，提高AR展示效果，提高用户体验。标识对象消失前后设置不同的虚拟对象展示效果，一方面增强互动性和趣味性，另一方面提示用户标识对象从当前场景图像中消失，提高用户体验。
本公开实施方式提供了一种AR设备，包括：
处理器；和
存储器，与处理器可通信连接，存储器存储有能够被处理器读取的计算机指令，计算机指令用于使处理器执行根据第一方面任一实施方式的方法。
本公开实施方式提供了一种存储介质，存储有计算机指令，计算机指令用于使计算机执行根据第一方面任一实施方式的方法。
具体而言，对于本公开示例的AR设备和存储介质相关实现方式，本领域技术人员参照前述实施方式可以理解并充分实施，本公开对此不再赘述。
本公开涉及增强现实领域，通过获取现实环境中的目标对象的图像信息，进而借助各类视觉相关算法实现对目标对象的相关特征、状态及属性进行检测或识别处理，从而得到与具体应用匹配的虚拟与现实相结合的AR效果。示例性的，目标对象可涉及与人体相关的脸部、肢体、手势、动作等，或者与物体相关的标识物、标志物，或者与场馆或场所相关的沙盘、展示区域或展示物品等。视觉相关算法可涉及视觉定位、SLAM、三维重建、图像注册、背景分割、对象的关键点提取及跟踪、对象的位姿或深度检测等。具体应用不仅可以涉及跟真实场景或物品相关的导览、导航、讲解、重建、虚拟效果叠加展示等交互场景，还可以涉及与人相关的特效处理，比如妆容美化、肢体美化、特效展示、虚拟模型展示等交互场景。可通过卷积神经网络，实现对目标对象的相关特征、状态及属性进行检测或识别处理。上述卷积神经网络是基于深度学习框架进行模型训练而得到的网络模型。
上述实施方式仅仅是为清楚地说明所作的举例，而并非对实施方式的限定。对于所属领域的普通技术人员来说，在上述说明的基础上还可以做出其它不同形式的变化或变动。这里无需也无法对所有的实施方式予以穷举。而由此所引伸出的显而易见的变化或变动仍处于本公开创造的保护范围之中。

Claims (10)

  1. 一种增强现实AR场景的展示方法，其特征在于，所述方法包括：
    响应于从当前场景的第一图像中识别到标识对象，根据所述标识对象对应的虚拟对象在世界坐标系中的第一位姿数据，确定所述虚拟对象在AR设备的屏幕坐标系的第二位姿数据；
    基于所述第二位姿数据，在所述AR设备展示所述虚拟对象；
    在所述虚拟对象的展示过程中，响应于从所述当前场景的第二图像中未识别到所述标识对象，基于预先设置的在所述屏幕坐标系中的目标位姿数据，在所述AR设备的屏幕上展示所述虚拟对象。
  2. 根据权利要求1所述的展示方法，其特征在于，在所述虚拟对象的展示过程中，响应于从所述当前场景的第二图像中未识别到所述标识对象，基于所述预先设置的在所述屏幕坐标系中的所述目标位姿数据，在所述AR设备的屏幕上展示所述虚拟对象，包括：
    在所述虚拟对象的展示过程中，响应于从所述当前场景的第二图像中未识别到所述标识对象，确定当前已展示进度；
    基于所述目标位姿数据和所述已展示进度，在所述AR设备的屏幕上继续展示所述虚拟对象。
  3. 根据权利要求1或2所述的展示方法，其特征在于，还包括：
    响应于从所述当前场景的所述第二图像之后采集的第三图像中识别到所述标识对象，根据所述标识对象对应的所述虚拟对象在所述世界坐标系中的第三位姿数据，确定所述虚拟对象在所述AR设备的所述屏幕坐标系的第四位姿数据；
    基于所述第四位姿数据，在所述AR设备展示所述虚拟对象。
  4. 根据权利要求1至3任一所述的展示方法，其特征在于，还包括：
    在从所述当前场景的所述第一图像中识别到所述标识对象的情况下，以第一展示效果展示所述虚拟对象；
    在从所述当前场景的所述第二图像中未识别到所述标识对象的情况下，以第二展示效果展示所述虚拟对象，所述第二展示效果不同于所述第一展示效果。
  5. 根据权利要求1至4任一所述的展示方法，其特征在于，所述目标位姿数据包括针对所述虚拟对象的展示位置数据和展示姿态数据；所述基于预先设置的在所述屏幕坐标系中的目标位姿数据，在所述AR设备的屏幕上展示所述虚拟对象，包括：
    在所述展示位置数据所指示的屏幕中的位置处，以所述展示姿态数据所指示的姿态展示所述虚拟对象。
  6. 根据权利要求1至5任一项所述的展示方法，其特征在于，根据所述标识对象对应的所述虚拟对象在所述世界坐标系中的所述第一位姿数据，确定所述虚拟对象在所述AR设备的所述屏幕坐标系的所述第二位姿数据，包括：
    根据所述虚拟对象在所述世界坐标系中的预设位姿，确定所述第一位姿数据；所述世界坐标系为根据所述标识对象的特征确定的；
    根据所述第一位姿数据，以及所述世界坐标系与所述AR设备的所述屏幕坐标系的映射关系，确定所述虚拟对象在所述屏幕坐标系的所述第二位姿数据。
  7. 根据权利要求1至5任一项所述的展示方法，其特征在于，从所述当前场景的所述第一图像中识别到所述标识对象，包括：
    对所述当前场景的所述第一图像进行特征点提取，得到所述当前场景的所述第一图像包括的至少一个特征点对应的特征信息；
    将所述至少一个特征点对应的特征信息与预先存储的所述标识对象的特征点信息进行对比，识别所述当前场景的所述第一图像中的所述标识对象。
  8. 一种增强现实AR场景的展示装置，其特征在于，所述装置包括：
    第一位姿确定模块，被配置为响应于从当前场景的第一图像中识别到标识对象，根据所述标识对象对应的虚拟对象在世界坐标系中的第一位姿数据，确定所述虚拟对象在AR设备的屏幕坐标系的第二位姿数据；
    第一展示模块，被配置为基于所述第二位姿数据，在所述AR设备展示所述虚拟对象；
    第二展示模块，被配置为在所述虚拟对象的展示过程中，响应于从所述当前场景的第二图像中未识别到所述标识对象，基于预先设置的在所述屏幕坐标系中的目标位姿数据，在所述AR设备的屏幕上展示所述虚拟对象。
  9. 一种AR设备，其特征在于，包括：
    处理器；和
    存储器，与所述处理器可通信连接，所述存储器存储有能够被所述处理器读取的计算机指令，所述计算机指令用于使所述处理器执行根据权利要求1至7任一项所述的方法。
  10. 一种存储介质，其特征在于，存储有计算机指令，所述计算机指令用于使计算机执行根据权利要求1至7任一项所述的方法。
PCT/CN2022/134689 2021-12-31 2022-11-28 增强现实场景的展示 WO2023124691A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111673087.5 2021-12-31
CN202111673087.5A CN114299263A (zh) 2021-12-31 2021-12-31 增强现实ar场景的展示方法及装置

Publications (1)

Publication Number Publication Date
WO2023124691A1 true WO2023124691A1 (zh) 2023-07-06

Family

ID=80975553

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/134689 WO2023124691A1 (zh) 2021-12-31 2022-11-28 增强现实场景的展示

Country Status (2)

Country Link
CN (1) CN114299263A (zh)
WO (1) WO2023124691A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114299263A (zh) * 2021-12-31 2022-04-08 北京绵白糖智能科技有限公司 增强现实ar场景的展示方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111882674A (zh) * 2020-07-30 2020-11-03 北京市商汤科技开发有限公司 虚拟对象的调整方法、装置、电子设备及存储介质
CN112148189A (zh) * 2020-09-23 2020-12-29 北京市商汤科技开发有限公司 一种ar场景下的交互方法、装置、电子设备及存储介质
CN112882576A (zh) * 2021-02-26 2021-06-01 北京市商汤科技开发有限公司 Ar交互方法、装置、电子设备及存储介质
CN114299263A (zh) * 2021-12-31 2022-04-08 北京绵白糖智能科技有限公司 增强现实ar场景的展示方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111882674A (zh) * 2020-07-30 2020-11-03 北京市商汤科技开发有限公司 虚拟对象的调整方法、装置、电子设备及存储介质
CN112148189A (zh) * 2020-09-23 2020-12-29 北京市商汤科技开发有限公司 一种ar场景下的交互方法、装置、电子设备及存储介质
CN112882576A (zh) * 2021-02-26 2021-06-01 北京市商汤科技开发有限公司 Ar交互方法、装置、电子设备及存储介质
CN114299263A (zh) * 2021-12-31 2022-04-08 北京绵白糖智能科技有限公司 增强现实ar场景的展示方法及装置

Also Published As

Publication number Publication date
CN114299263A (zh) 2022-04-08

Similar Documents

Publication Publication Date Title
US9224237B2 (en) Simulating three-dimensional views using planes of content
US10496910B2 (en) Inconspicuous tag for generating augmented reality experiences
US11127210B2 (en) Touch and social cues as inputs into a computer
US9437038B1 (en) Simulating three-dimensional views using depth relationships among planes of content
US9591295B2 (en) Approaches for simulating three-dimensional views
US10295826B2 (en) Shape recognition device, shape recognition program, and shape recognition method
TW202119199A (zh) 虛擬鍵盤
US9041734B2 (en) Simulating three-dimensional features
WO2023124693A1 (zh) 增强现实场景的展示
US10037614B2 (en) Minimizing variations in camera height to estimate distance to objects
US9123272B1 (en) Realistic image lighting and shading
US20130174213A1 (en) Implicit sharing and privacy control through physical behaviors using sensor-rich devices
US9979946B2 (en) I/O device, I/O program, and I/O method
CN111742281B (zh) 用于针对显示在显示器上的第一内容提供根据外部对象的移动的第二内容的电子装置及其操作方法
WO2023124691A1 (zh) 增强现实场景的展示
US9449427B1 (en) Intensity modeling for rendering realistic images
US20150381970A1 (en) Input/output device, input/output program, and input/output method
WO2023124698A1 (zh) 增强现实场景的展示
US9911237B1 (en) Image processing techniques for self-captured images
WO2023245133A1 (en) Efficient multi-scale orb without image resizing
KR20240008370A (ko) 움직이는 객체들의 레이턴시를 최소화하기 위한 늦은 워핑
WO2019111052A2 (en) Inserting virtual objects in between two real objects in an augmented reality environment
Fan Mobile Room Schedule Viewer Using Augmented Reality

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22913949

Country of ref document: EP

Kind code of ref document: A1