WO2023124693A1 - 增强现实场景的展示 - Google Patents

增强现实场景的展示 Download PDF

Info

Publication number
WO2023124693A1
WO2023124693A1 PCT/CN2022/134712 CN2022134712W WO2023124693A1 WO 2023124693 A1 WO2023124693 A1 WO 2023124693A1 CN 2022134712 W CN2022134712 W CN 2022134712W WO 2023124693 A1 WO2023124693 A1 WO 2023124693A1
Authority
WO
WIPO (PCT)
Prior art keywords
display
virtual object
data
pose data
coordinate system
Prior art date
Application number
PCT/CN2022/134712
Other languages
English (en)
French (fr)
Inventor
李斌
欧华富
李颖楠
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Publication of WO2023124693A1 publication Critical patent/WO2023124693A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/20—Scenes; Scene-specific elements in augmented reality scenes

Definitions

  • the present disclosure relates to the technical field of augmented reality, in particular to a display of augmented reality scenes.
  • Augmented Reality (AR) technology superimposes a virtual model on the real world, thereby presenting the real environment and virtual objects in the same picture or space in real time.
  • AR Augmented Reality
  • Embodiments of the present disclosure provide a display method, device, AR device, and storage medium of an augmented reality AR scene.
  • an embodiment of the present disclosure provides a method for displaying an augmented reality AR scene, the method including: in response to identifying an identification object from the current scene image, based on the virtual object corresponding to the identification object of the AR device The first relative pose data in the world coordinate system, determine the display pose data of the virtual object; based on the display pose data, display the virtual object on the AR device; during the display process of the virtual object wherein, according to the second relative pose data of the virtual object and the identified object in the world coordinate system, determine the target display data of the virtual object; according to the display pose data and the target display data , displaying the virtual object on the AR device.
  • the embodiment of the present disclosure provides an augmented reality AR scene display device, the device includes: a first pose determination module, configured to respond to identifying an identification object from the current scene image, based on the AR device
  • the first relative pose data of the virtual object corresponding to the identified object in the world coordinate system determines the display pose data of the virtual object
  • the first display module is configured to, based on the display pose data, in The AR device displays the virtual object
  • the display data determination module is configured to, during the display process of the virtual object, according to the second relative position between the virtual object and the identification object in the world coordinate system
  • the pose data is used to determine the target display data of the virtual object
  • the second display module is configured to display the virtual object on the AR device according to the display pose data and the target display data.
  • the embodiment of the present disclosure provides an augmented reality AR device, including: a processor; and a memory, communicably connected to the processor, the memory stores computer instructions that can be read by the processor , the computer instructions are used to cause the processor to execute the method according to any implementation manner of the first aspect.
  • an embodiment of the present disclosure provides a storage medium storing computer instructions for causing a computer to execute the method according to any embodiment of the first aspect.
  • the presentation method of the embodiment of the present disclosure includes, in response to identifying the identification object from the current scene image, determining the display position of the virtual object based on the first relative pose data of the virtual object corresponding to the AR device and the identification object in the world coordinate system Pose data, based on the display pose data, the virtual object is displayed on the AR device, and the virtual object is displayed based on the first relative pose, so that the display effect of the virtual object is more realistic and the display effect is improved.
  • the virtual object can display different display effects in real time based on the relative pose changes between the logo object and the virtual object, thereby increasing the interactivity of the AR scene and improving the display effect of the virtual image.
  • Fig. 1 is a schematic structural diagram of an AR device according to some implementations of the present disclosure.
  • FIGS. 2a and 2b are schematic structural diagrams of an AR device according to some implementations of the present disclosure.
  • Fig. 3 is a structural block diagram of an AR device according to some implementations of the present disclosure.
  • FIG. 4 is a flowchart illustrating a method in some embodiments according to the present disclosure.
  • Figure 5 is a schematic diagram illustrating a method in some embodiments according to the present disclosure.
  • FIG. 6 is a flowchart illustrating a method in some embodiments according to the present disclosure.
  • Figure 7 is a flow chart illustrating a method in some embodiments according to the present disclosure.
  • FIG. 8 is a flowchart illustrating a method in some embodiments according to the present disclosure.
  • FIG. 9 is a flowchart illustrating a method in some embodiments according to the present disclosure.
  • Figure 10 is a schematic diagram illustrating a method in some embodiments according to the present disclosure.
  • Fig. 11 is a structural block diagram of a display device according to some embodiments of the present disclosure.
  • Displaying AR effects based on markers is a way to display virtual objects in augmented reality scenes.
  • This method requires pre-made markers, such as pre-made markers drawn on the surface of items based on Hamming Code technology.
  • Graphics, etc., the preset graphics can be, for example, cartoon images, two-dimensional codes, barcodes, etc.
  • the display of the virtual object is realized based on the relative pose of the AR device and the virtual object. For example, when the AR device is far away from the virtual object, the virtual object displayed on the display screen becomes smaller; When rotating, the virtual object displayed on the display screen rotates accordingly.
  • This display method is not interactive enough and the display effect is not good.
  • Embodiments of the present disclosure provide a display method, device, AR device, and storage medium for an augmented reality AR scene, aiming at improving the display effect of virtual objects in the AR scene and improving user experience.
  • the embodiment of the present disclosure provides a method for displaying an augmented reality AR scene, and the method can be applied to an AR device.
  • the AR device described in the embodiments of the present disclosure can be any terminal device with AR functions, such as head-mounted devices such as AR glasses and AR helmets, or handheld devices such as smart phones and tablet computers, or smart phones. Wrist-worn devices such as watches and smart bracelets, etc., are not limited in this disclosure.
  • the AR device described in the present disclosure is described by taking AR glasses as an example.
  • the AR device 100 includes a main body 101, and the main body 101 includes a bracket structure for wearing and supporting.
  • the display screen 110 is arranged on the main body 101 , and after the user wears the AR device 100 , the display screen 110 can be located directly in front of the user's eyes, so that the user can easily observe the images displayed on the display screen 110 .
  • the AR device 100 also includes a camera 120, which is used to collect images of the current scene.
  • the AR device 100 also includes a processor disposed inside the main body 101 , and the processor is used to superimpose the current scene image captured by the camera 120 with the virtual object, and display it on the display screen 110 .
  • the AR device described in the present disclosure is described by taking a smart phone as an example.
  • the AR device 100 includes a housing 102, which is a shell structure for supporting the main body of the mobile phone.
  • the display screen 110 is arranged on the front of the casing 102, and the camera 120 is arranged on the front and/or the back of the casing 102.
  • the camera 120 is arranged on the back of the casing.
  • the camera 120 is used to collect the current scene image, so that the processor of the smart phone superimposes the current scene image collected by the camera 120 with the virtual object, and displays it on the display screen 110 .
  • FIG. 3 shows a structural block diagram of an AR device in the present disclosure.
  • the AR device in an embodiment of the present disclosure will be further described below in conjunction with FIG. 3 .
  • the AR device 100 includes a processor 130 , a memory 140 , a camera 120 , a display screen 110 and an IMU (Inertial Measurement Unit, inertial measurement unit) sensor 160 .
  • IMU Inertial Measurement Unit, inertial measurement unit
  • the processor 130 , the memory 140 , the camera 120 , the display screen 110 and the IMU sensor 160 establish a communicable connection between any of them through the bus 150 .
  • the processor 130 can be any type of processor with one or more processing cores. It can perform single-threaded or multi-threaded operations, and is used to parse instructions to perform operations such as obtaining data, performing logical operation functions, and delivering operation processing results.
  • the memory 140 may include a non-volatile computer-readable storage medium, such as at least one magnetic disk storage device, a flash memory device, a distributed storage device located remotely with respect to the processor 130, or other non-volatile solid-state storage devices.
  • the memory may have a program storage area for storing non-volatile software programs, non-volatile computer-executable programs and modules, which are invoked by the processor 130 to make the processor 130 execute one or more method steps below.
  • the memory 140 may also include a storage part such as a volatile random storage medium or a hard disk, as a data storage area for storing calculation processing results and data output by the processor 130 .
  • the camera 120 is used to collect the current scene image, so that the processor 130 can perform fusion processing for the virtual object according to the collected current scene image, so as to realize the AR display effect.
  • the camera 120 can be any type of camera suitable for implementation, such as a black and white camera, an RGB camera, etc.
  • the camera 120 can be divided into a monocular camera and a binocular camera according to the number of cameras, which is not limited in the present disclosure. .
  • the display screen 110 is configured to receive the display information issued by the processor 130 , so as to present a corresponding display image on the display screen 110 according to the display information, so that the user can observe the AR display effect through the display screen 110 .
  • the display screen 110 may be any display screen type suitable for implementation, such as an LCD (Liquid Crystal Display, liquid crystal display) display screen, an OLED (Organic Light-Emitting Diode, organic light-emitting semiconductor) display screen, etc., This disclosure does not limit this.
  • the IMU sensor 160 is mainly used to detect and measure the acceleration and rotation of the AR device itself. Its basic principle is to use the law of inertia to measure the acceleration and rotation. It can calculate the pose of the AR device itself in real time based on the sensor signal.
  • the IMU sensor 160 may include, for example, a three-axis or six-axis gyroscope, an acceleration sensor, and the like.
  • the display method of the AR scene in the example of the present disclosure includes:
  • the current scene image refers to a scene image captured by a camera of an AR device in real time, or a scene image collected by a camera device and transmitted to the AR device.
  • the camera 120 collects scene images within the viewing range, and displays the collected scene images on the display screen 110 through data processing.
  • the images observed by the user in real time on the display screen 110 are The current scene image. It can be understood that as the pose of the AR device changes, the collected current scene image changes accordingly, that is, the current scene image is a real-time image.
  • the identification object refers to the marker used to trigger the display of the AR effect, that is, the above-mentioned Marker.
  • the specific presentation mode of the identification object may be, for example, a QR code, a preset graphic, etc., which is not limited in the present disclosure.
  • the identification object 210 is a cartoon graphic drawn on the surface of the postcard 200, and the cartoon graphic can be drawn based on Hamming code technology, so that when the AR device captures an image containing the cartoon graphic, It can recognize and analyze cartoon graphics.
  • the process of drawing and analyzing Hamming codes can be understood and fully implemented by those skilled in the art by referring to related technologies, which is not limited in the present disclosure.
  • the identification object is a preset graphic drawn on the surface of the object.
  • the AR device captures the scene image including the preset graphic, it can extract and identify the feature points of the scene image through image detection technology, so that The preset image is recognized from the scene image.
  • image detection technology so that The preset image is recognized from the scene image.
  • the identification object is associated with the corresponding virtual object.
  • a model database including at least one virtual object can be pre-established, and the model database includes the virtual object and the identification object associated with the virtual object.
  • the virtual object corresponding to the identification object can be determined by searching the model database.
  • a virtual object is represented as a virtual image rendered and presented on the display screen of an AR device.
  • the display effect of the interaction between the virtual object and the real scene can be presented.
  • the AR device detects and recognizes the collected current scene image through the image detection technology, and if the identified object is identified from the current scene image, the virtual object corresponding to the identified object can be determined.
  • the AR device When the AR device recognizes the identified object from the current scene image, it can display the corresponding virtual object on the AR device based on Simultaneous Localization And Mapping (SLAM) technology.
  • SLAM Simultaneous Localization And Mapping
  • the SLAM technology can realize the spatial positioning of the AR device and the map construction for the current scene, so that the superimposed display effect of the virtual object and the real scene can be rendered on the current scene image displayed by the AR device, making the display effect of the virtual object more realistic.
  • the world coordinate system can be pre-constructed based on the identified object, and then according to the real-time pose data of the virtual object in the world coordinate system and the real-time pose data of the AR device in the world coordinate system, determine the relationship between the virtual object and the AR
  • the relative pose data between devices that is, the first relative pose data described in this disclosure, and then based on the mapping relationship between the screen coordinate system and the world coordinate system, determine the position of the virtual object in the screen coordinate system according to the first relative pose data.
  • Display pose data in .
  • displaying the pose data includes displaying relevant data of the virtual object in the screen coordinate system of the AR device, so that after the display pose data is obtained at S410, the virtual object can be displayed on the screen corresponding to the display pose data. position so that the user can observe the virtual object at the corresponding position on the screen.
  • image detection technology is used to detect the pose data of the identified object in the current scene image, so as to determine the relationship between the identified object and the virtual object in the current scene image.
  • the relative pose of the object in the world coordinate system that is, the second relative pose data described in this disclosure.
  • corresponding display data may be preset for different second relative pose data in the world coordinate system between the virtual object and the identification object, and different display data correspond to different display effects. Therefore, during the display process of the virtual object, the relative pose change between the virtual object and the marked object is detected in real time, that is, the change of the second relative pose data, and the corresponding target display is determined in real time according to the current second relative pose data The AR device presents the corresponding display effect on the display screen according to the target display data.
  • the second relative pose data may include the relative distance between the virtual object and the identified object in the world coordinate system, and different relative distances correspond to different presentation data.
  • the display effect corresponding to different display data is: the virtual object presents a display effect with gradually increasing transparency. Therefore, the relative distance between the virtual object and the identification object can be detected in real time, and corresponding transparency data, that is, target display data, can be determined according to the current relative distance.
  • the second relative pose data is not limited to the relative distance between the virtual object and the identified object, but can also be any other suitable relative pose, such as relative angle, relative posture, etc., and this disclosure does not make any limit.
  • the display data is not limited to the transparency data of the virtual object, and can also be any other display data suitable for implementation, such as blur effect, size, brightness data, etc., which is not limited in the present disclosure.
  • the target presentation data indicates the presentation effect rendered for the virtual object
  • the presentation pose data indicates the presentation pose of the virtual object in the screen coordinate system. Therefore, after the target display data is determined, the virtual object can be rendered and displayed based on the target display data and the display pose data.
  • the target display data is "display the virtual object with 50% transparency", so that the AR device can re-render the virtual object, so that the virtual object can be displayed on the screen of the AR device with a transparency of 50%. Transparency is displayed.
  • the target display data is "display the virtual object at a size of 50%", so that the AR device can re-render the virtual object, so that the virtual object is scaled to 50% on the screen of the AR device. % size for display.
  • target display data is not limited to the above examples, and this disclosure will not enumerate them.
  • the process of identifying an object from the current scene image includes:
  • S610 Perform feature point extraction on the current scene image to obtain feature information corresponding to at least one feature point included in the current scene image.
  • the AR device 100 can collect the current scene image through the camera 120. It can be understood that the current scene image can be a single frame image collected by the camera, or a multi-frame image in the video stream collected by the camera. There is no limit to this.
  • the detection area including the object to be detected in the current scene image can be located by an image detection algorithm, and then one or more feature points of the object to be detected can be extracted in the detection area.
  • the image detection algorithm can be used to first determine the area of the postcard 200 as the area to be tested, and then extract the feature points of the "Little Dragon Man” and "Auspicious Cloud” included on the postcard 200 to obtain The characteristic information of each object to be tested.
  • the feature information of the object to be tested can be compared with the feature point information of the pre-stored identification object. If the similarity between the two is greater than the preset threshold, it means that the object to be tested is an identification object; otherwise, if the similarity is not greater than the preset threshold, it means that the object to be tested is not the identified object.
  • the feature point information described in the embodiments of the present disclosure may include texture feature value, RGB feature value, gray value and other information that can identify the feature point feature corresponding to the feature point, which is not limited in the present disclosure.
  • the virtual object corresponding to the identification object and the display pose data required for displaying the virtual object can be further obtained. The following will describe the embodiment in conjunction with FIG. 7 .
  • the process of determining the presentation pose data of the virtual object includes:
  • S710 According to the initial relative poses of the virtual object and the AR device, determine first relative pose data through real-time localization and mapping (SLAM).
  • SLAM real-time localization and mapping
  • S720 Determine display pose data of the virtual object in the screen coordinate system according to the first relative pose data and the mapping relationship between the world coordinate system and the screen coordinate system of the AR device.
  • the AR device when displaying a virtual object based on SLAM, it is necessary to map the virtual object and the AR device into the same coordinate system, so that when the relative motion of the two poses occurs, the AR device can use the IMU sensor 160 to calculate and determine the two coordinates in real time.
  • the real-time relative pose of the subject, the coordinate system is the world coordinate system described in this disclosure.
  • features of the marked object can be extracted in advance based on image detection technology to obtain at least one preset key point of the marked object, and a world coordinate system can be established based on the at least one preset key point.
  • a preset initial pose of each virtual object in the world coordinate system is determined, that is, a first initial pose.
  • the first initial pose may include initial positions and poses of each virtual object.
  • the initial pose of the AR device in the world coordinate system that is, the second initial pose
  • the second initial pose can also include the initial position and pose of the AR device .
  • the initial pose of the AR device in the world coordinate system can be determined from the pose of the IMU sensor 160 when the AR device 100 captures the first frame of the scene image.
  • the initial pose of the AR device and the initial pose of the virtual object can be Get the initial relative pose of the virtual object and the AR device.
  • the relative position of the virtual object and the AR device is determined according to the position difference between the two, and the relative posture of the virtual object and the AR device is determined according to the initial posture difference between the virtual object and the AR device.
  • the IMU sensor 160 may detect that the pose (position and attitude) of the AR device 100 changes.
  • the gyroscope can determine the posture of the AR device after the pose changes according to the angle deviation between the AR device in the three-axis direction and the initial posture, and the acceleration sensor can detect the mobile acceleration during the movement of the AR device, and use the inertial navigation solution method Get the position after movement, so as to determine the pose data of the AR device after movement.
  • the aforementioned initial relative pose is updated accordingly to obtain the relative pose between the updated pose of the AR device and the pose of the virtual object data, that is, the first relative pose data.
  • the world coordinate system can be mapped to the camera coordinate system of the AR device through rotation and translation, and then mapped from the camera coordinate system to the screen coordinate system of the AR device.
  • the AR device determines the first relative pose data between itself and the virtual object in real time, it can map the first relative pose data to the screen coordinate system of the AR device according to the above mapping relationship, and determine that the virtual object is in the screen coordinate system display pose data. Then the AR device renders the display on the display screen of the AR device according to the display pose data, so that the user can watch the real-time pose change of the virtual object.
  • the virtual object when the user wears AR glasses such as shown in FIG. 1 and gradually moves away from the initial position, based on the implementation shown in FIG. When approaching the initial position, the virtual object can be observed on the display screen 110 to gradually become larger.
  • the display effect of the embodiments of the present disclosure conforms to the "far smaller and near larger" when viewed by human eyes in a real scene, so the display of virtual objects is more realistic.
  • the virtual object is displayed based on the SLAM method, so that the display effect of the virtual object is more realistic and user experience is improved.
  • the presentation method of the example of the present disclosure includes:
  • S830 Determine an initial relative pose of the virtual object and the AR device according to the preset initial pose of the virtual object in the world coordinate system and the initial pose of the AR device in the world coordinate system.
  • the logo object 210 is a cartoon image drawn on the surface of the postcard 200.
  • the AR device detects and recognizes that the current scene image contains a logo
  • a world coordinate system can be established based on the preset key points that identify the object.
  • the preset key point may be the center point of the identified object, and the center point of the identified object is used as the origin O of the coordinate system, and the direction parallel to the short side of the postcard 200 and passing through the origin O is used as the
  • the X-axis takes the direction parallel to the long side of the postcard 200 and passes through the origin O as the Y-axis, and takes the direction perpendicular to the X-axis and Y-axis and passes through the origin O as the Z-axis, thereby establishing the world coordinate system O-XYZ.
  • the process of establishing the world coordinate system in FIG. 5 is only an example of the implementation of the present disclosure.
  • the world coordinate system can also be established according to other preset key points, for example, the vertex in the lower left corner of the postcard 200 can be used as
  • the coordinate origin O establishes a coordinate system, which is not limited in the present disclosure.
  • the initial pose of the virtual object in the world coordinate system can be preset, that is, the preset initial pose, and the preset initial pose can include the initial position and pose of the virtual object.
  • the preset initial pose of the virtual object includes: standing facing the -y direction at the origin O of the world coordinate system, that is, the preset initial pose includes the position (origin O) and posture of the virtual object (stand facing -y direction).
  • the initial pose of the AR device in the world coordinate system can be obtained when the AR device captures the first frame of the scene image, and the initial pose can also include the position and posture of the AR device.
  • the pose of the IMU sensor 160 may be determined as the initial pose of the AR device in the world coordinate system when the AR device 100 captures the first frame of the scene image.
  • the initial relative pose of the virtual object and the AR device can be obtained based on the two.
  • the relative position data of the virtual object and the AR device are determined according to the initial position difference between the two
  • the relative posture data of the virtual object and the AR device are determined according to the initial posture difference between the virtual object and the AR device.
  • the relative position data and the relative posture data constitute the virtual object and AR device.
  • the initial relative pose of the AR device is determined according to the initial position difference between the two.
  • the process of determining the target display data of the virtual object includes:
  • the current scene image can be shown in Figure 5.
  • the AR device recognizes the identification object 210 in the current scene image ( Figure 5), it can realize the display of the virtual object through the foregoing implementation.
  • the display effect is shown in Figure 10.
  • the pose of the marked object can be detected simultaneously based on the image detection technology. It can be understood that during the display process of the virtual object, the pose of the marked object may change. For example, the user moves the postcard 200, so that The pose of the identified object in the current scene image detected by the AR device changes.
  • the first pose data of the virtual object in the world coordinate system may be acquired, for example, the aforementioned preset initial pose of the virtual object in the world coordinate system is used as the first pose data.
  • the AR device can detect the second pose data of the marked object in the world coordinate system.
  • the detection frame of the postcard 200 can be determined based on image detection technology, and the second pose data of the marked object can be determined according to the difference between the detection frame of the postcard 200 before and after the pose change. pose data.
  • the relative pose between the virtual object and the marked object can be determined according to the first pose data and the second pose data, that is, the The second relative pose data.
  • the corresponding relationship between the second relative pose data and the display data may be established in advance based on the second relative pose data.
  • the second relative pose data includes the relative distance between the virtual object and the identified object in the world coordinate system
  • the corresponding display data includes the transparency of the virtual object
  • the second relative pose data includes the relative posture (such as relative angle) of the virtual object and the identified object in the world coordinate system
  • the corresponding display data includes the brightness of the virtual object
  • the second relative pose data also includes the relative distance and relative pose of the virtual object and the identified object in the world coordinate system
  • the corresponding display data includes the transparency and brightness of the virtual object, so that the pre-established display effect
  • the pre-established display effect relationship may also include other methods, for example, the display data may also include color, brightness, transparency, exposure Or display one or more types of data in the perspective, which is not limited in the present disclosure.
  • the display effect relationship is shown in Table 3 above, and the detected second relative pose data includes: the relative distance is located in L1-L2, and the relative angle is located in ⁇ 2- ⁇ 3. Therefore, according to Table 3, it can be determined that the target display data is: transparency 25% and brightness 50%.
  • the virtual object can be rendered in real time based on the target display data and the previously determined display pose data, so that the user can watch the display effect of the virtual object corresponding to the target display data on the display screen of the AR device .
  • an embodiment of the present disclosure provides an AR scene display device, which can be applied to an AR device.
  • the AR device may be, for example, the AR glasses shown in FIG. 1 .
  • the AR device may be, for example, a smart phone as shown in Fig. 2a and Fig. 2b. This disclosure does not limit this.
  • the AR scene display device of the example of the present disclosure includes:
  • the first pose determining module 10 is configured to, in response to identifying the identified object from the current scene image, based on the first relative pose data of the virtual object corresponding to the AR device and the identified object in the world coordinate system, determine the position of the virtual object Display pose data;
  • the first display module 20 is configured to display the virtual object on the AR device based on the display pose data
  • the display data determination module 30 is configured to determine the target display data of the virtual object according to the second relative pose data of the virtual object and the identification object in the world coordinate system during the display process of the virtual object;
  • the second display module 40 is configured to display the virtual object on the AR device according to the display pose data and the target display data.
  • the presentation data determination module 30 is configured to:
  • the first pose data of the virtual object in the world coordinate system and the second pose data of the identified object in the world coordinate system are obtained;
  • target display data corresponding to the second relative pose data is determined.
  • the second relative pose data includes the relative position and/or relative pose between the virtual object and the identified object.
  • the display data includes one or more data of color, brightness, transparency, or display angle of the virtual object.
  • the first pose determination module 10 is configured to:
  • the initial relative pose of the virtual object and the AR device determine the first relative pose data through real-time positioning and mapping;
  • the display pose data of the virtual object in the screen coordinate system is determined.
  • the first pose determination module 10 is configured to:
  • the feature information corresponding to the at least one feature point is compared with the pre-stored feature point information of the identified object to identify the identified object in the current scene image.
  • An embodiment of the present disclosure provides an AR device, including:
  • the memory is communicably connected to the processor, and the memory stores computer instructions that can be read by the processor, and the computer instructions are used to make the processor execute the method according to any implementation manner of the first aspect.
  • Embodiments of the present disclosure provide a storage medium storing computer instructions, which are used to make a computer execute the method according to any embodiment of the first aspect.
  • This disclosure relates to the field of augmented reality.
  • acquiring the image information of the target object in the real environment and then using various visual correlation algorithms to detect or identify the relevant features, states and attributes of the target object, and thus obtain the image information that matches the specific application.
  • AR effect combining virtual and reality.
  • the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places.
  • Vision-related algorithms can involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc.
  • Specific applications can not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlays and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual Interactive scenarios such as model display.
  • the relevant features, states and attributes of the target object can be detected or identified through the convolutional neural network.
  • the above-mentioned convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Computer Hardware Design (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Computer Graphics (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

本公开涉及一种AR场景的展示方法，包括响应于从当前场景图像中识别到标识对象，基于根据AR设备与虚拟对象的第一相对位姿数据确定的展示位姿数据，在AR设备展示虚拟对象，在虚拟对象的展示过程中，根据虚拟对象与标识对象的第二相对位姿数据，确定虚拟对象的目标展示数据，根据展示位姿数据和目标展示数据在AR设备展示虚拟对象。

Description

增强现实场景的展示
相关申请的交叉引用
本申请要求在2021年12月31日提交至中国专利局、申请号为CN2021116671415的中国专利申请的优先权，其全部内容通过引用结合在本公开中。
技术领域
本公开涉及增强现实技术领域，具体涉及一种增强现实场景的展示。
背景技术
增强现实(Augmented Reality，AR)技术通过将虚拟模型叠加到真实世界中，从而将真实的环境和虚拟的对象实时地在同一个画面或者空间呈现。对于AR场景，如何丰富虚拟对象与真实场景的交互，优化展示效果是主要的研究方向之一。
发明内容
本公开实施方式提供了一种增强现实AR场景的展示方法、装置、AR设备以及存储介质。
第一方面，本公开实施方式提供了一种增强现实AR场景的展示方法，所述方法包括：响应于从当前场景图像中识别到标识对象，基于AR设备与所述标识对象对应的虚拟对象在世界坐标系中的第一相对位姿数据，确定所述虚拟对象的展示位姿数据；基于所述展示位姿数据，在所述AR设备展示所述虚拟对象；在所述虚拟对象的展示过程中，根据所述虚拟对象与所述标识对象在所述世界坐标系中的第二相对位姿数据，确定所述虚拟对象的目标展示数据；根据所述展示位姿数据和所述目标展示数据，在所述AR设备展示所述虚拟对象。
第二方面，本公开实施方式提供了一种增强现实AR场景的展示装置，所述装置包括：第一位姿确定模块，被配置为响应于从当前场景图像中识别到标识对象，基于AR设备与所述标识对象对应的虚拟对象在世界坐标系中的第一相对位姿数据，确定所述虚拟对象的展示位姿数据；第一展示模块，被配置为基于所述展示位姿数据，在所述AR设备展示所述虚拟对象；展示数据确定模块，被配置为在所述虚拟对象的展示过程中，根据所述虚拟对象与所述标识对象在所述世界坐标系中的第二相对位姿数据，确定所述虚拟对象的目标展示数据；第二展示模块，被配置为根据所述展示位姿数据和所述目标展示数据，在所述AR设备展示所述虚拟对象。
第三方面，本公开实施方式提供了一种增强现实AR设备，包括：处理器；和存储器，与所述处理器可通信连接，所述存储器存储有能够被所述处理器读取的计算机指令，所述计算机指令用于使所述处理器执行根据第一方面任一实施方式所述的方法。
第四方面，本公开实施方式提供了一种存储介质，存储有计算机指令，所述计算机指令用于使计算机执行根据第一方面任一实施方式所述的方法。
本公开实施方式的展示方法，包括响应于从当前场景图像中识别到标识对象，基于AR设备与标识对象对应的虚拟对象在世界坐标系中的第一相对位姿数据，确定虚拟对象的展示位姿数据，基于展示位姿数据在AR设备展示虚拟对象，基于第一相对位姿来展示虚拟对象，使得虚拟对象的展示效果更加逼真，提高展示效果。并且在虚拟对象的展示过程中，根据虚拟对象与标识对象在世界坐标系中的第二相对位姿数据，确定虚拟 对象的目标展示数据，根据展示位姿数据和目标展示数据，在AR设备展示虚拟对象，可以基于标识对象与虚拟对象的相对位姿变化，实时展示不同的展示效果，从而增加AR场景的互动性，提高虚拟形象的展示效果。
附图说明
为了更清楚地说明本公开具体实施方式或相关技术中的技术方案，下面将对具体实施方式或相关技术的描述中所需要使用的附图作简单地介绍，下面描述中的附图是本公开的一些实施方式，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是根据本公开一些实施方式中AR设备的结构示意图。
图2a和2b是根据本公开一些实施方式中AR设备的结构示意图。
图3是根据本公开一些实施方式中AR设备的结构框图。
图4是根据本公开一些实施方式中展示方法的流程图。
图5是根据本公开一些实施方式中展示方法的原理图。
图6是根据本公开一些实施方式中展示方法的流程图。
图7是根据本公开一些实施方式中展示方法的流程图。
图8是根据本公开一些实施方式中展示方法的流程图。
图9是根据本公开一些实施方式中展示方法的流程图。
图10是根据本公开一些实施方式中展示方法的原理图。
图11是根据本公开一些实施方式中展示装置的结构框图。
具体实施方式
下面将结合附图对本公开的技术方案进行清楚、完整地描述，所描述的实施方式是本公开一部分实施方式，而不是全部的实施方式。基于本公开中的实施方式，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施方式，都属于本公开保护的范围。此外，下面所描述的本公开不同实施方式中所涉及的技术特征在彼此不冲突的情况下可以相互结合。
基于标识对象(Marker)进行AR效果展示是增强现实场景中对虚拟对象的一种展示方式，这种方式需要预先制作好的标识对象，例如基于汉明码(Hamming Code)技术绘制在物品表面的预设图形等，预设图形可以是例如卡通形象、二维码、条形码等。AR设备通过摄像头识别到当前场景中出现该Marker时，可在当前场景图像中叠加对应的虚拟对象，实现虚拟对象与现实场景叠加的展示效果。
在虚拟对象的展示过程中，基于AR设备与虚拟对象的相对位姿实现虚拟对象的展示，例如当AR设备远离虚拟对象时，显示屏所展示的虚拟对象随之变小；又例如当AR设备转动时，显示屏所展示的虚拟对象随之转动。这种展示方式互动性不足，展示效果不佳。
本公开实施方式提供了一种增强现实AR场景的展示方法、装置、AR设备以及存储介质，旨在提高AR场景中虚拟对象的展示效果，提高用户体验。
本公开实施方式提供了一种增强现实AR场景的展示方法，该方法可以应用于AR 设备。可以理解，本公开实施方式所述的AR设备，可以是任何具有AR功能的终端设备，例如AR眼镜、AR头盔等头戴式设备，又例如智能手机、平板电脑等手持式设备，再例如智能手表、智能手环等腕戴式设备等等，本公开对此不作限制。
如图1所示，在一个示例中，本公开所述的AR设备以AR眼镜为例进行说明。AR设备100包括主体101，主体101包括实现佩戴及支撑的支架结构。显示屏110设置在主体101上，用户在佩戴AR设备100之后，显示屏110可以位于用户眼睛的正前方，从而便于用户观察到显示屏110上显示的画面。AR设备100还包括摄像头120，摄像头120用于采集当前场景图像。AR设备100还包括设于主体101内部的处理器，处理器用于将摄像头120采集的当前场景图像与虚拟对象进行叠加之后，在显示屏110上展示。
如图2a所示，在一个示例中，本公开所述的AR设备以智能手机为例进行说明。AR设备100包括壳体102，壳体102为实现手机主体支撑的外壳结构。显示屏110设置在壳体102正面，摄像头120设置在壳体102的正面和/或背面，例如图2b所示，摄像头120设于壳体的背面。摄像头120用于采集当前场景图像，从而智能手机的处理器将摄像头120采集的当前场景图像与虚拟对象进行叠加之后，在显示屏110上展示。
当然可以理解，AR设备的具体设备类型不局限于上述示例的AR眼镜和智能手机，本公开对此不再赘述。图3中示出了本公开AR设备的结构框图，下面结合图3对本公开实施方式的AR设备进一步说明。
如图3所示，在一些实施方式中，AR设备100包括处理器130、存储器140、摄像头120、显示屏110以及IMU(Inertial Measurement Unit，惯性测量单元)传感器160。
处理器130、存储器140、摄像头120、显示屏110以及IMU传感器160通过总线150建立任意两者之间的可通信连接。
处理器130可以为任何类型，具备一个或者多个处理核心的处理器。其可以执行单线程或者多线程的操作，用于解析指令以执行获取数据、执行逻辑运算功能以及下发运算处理结果等操作。
存储器140可包括非易失性计算机可读存储介质，例如至少一个磁盘存储器件、闪存器件、相对于处理器130远程设置的分布式存储设备或者其他非易失性固态存储器件。存储器可以具有程序存储区，用于存储非易失性软件程序、非易失性计算机可执行程序以及模块，供处理器130调用以使处理器130执行下文一个或者多个方法步骤。存储器140还可以包括易失性随机存储介质、或者硬盘等存储部分，作为数据存储区，用以存储处理器130下发输出的运算处理结果及数据。
摄像头120用于采集当前场景图像，从而处理器130可以根据采集到的当前场景图像进行针对虚拟对象的融合处理，实现AR展示效果。在本公开实施方式中，摄像头120可以是任何适于实施的摄像头类型，例如黑白摄像头、RGB摄像头等，摄像头120按照摄像头的数量可以分为单目摄像头和双目摄像头，本公开对此不作限制。
显示屏110用于接收处理器130下发的显示信息，从而根据显示信息在显示屏110上呈现出对应的显示图像，以使用户可以通过显示屏110观察到AR展示效果。在本公开实施方式中，显示屏110可以是任何适于实施的显示屏类型，例如LCD(Liquid Crystal Display，液晶显示)显示屏、OLED(Organic Light-Emitting Diode，有机发光半导体)显示屏等，本公开对此不作限制。
IMU传感器160主要用来检测和测量AR设备自身的加速度与旋转运动，其基本原理是利用惯性定律实现加速度与旋转运动的测量，可以基于传感器信号实时对AR设备 自身位姿进行解算。在一些实施方式中，IMU传感器160可以包括例如三轴或六轴陀螺仪、加速度传感器等。
基于上述AR设备结构，下面对本公开实施方式的AR场景的展示方法进行说明。
如图4所示，在一些实施方式中，本公开示例的AR场景的展示方法包括：
S410、响应于从当前场景图像中识别到标识对象，基于AR设备与标识对象对应的虚拟对象在世界坐标系中的第一相对位姿数据，确定虚拟对象的展示位姿数据。
具体而言，当前场景图像是指利用AR设备的摄像头实时采集到的场景图像，或者是通过摄像设备采集并传输至AR设备的场景图像。例如图1所示的AR眼镜中，通过摄像头120采集取景范围内的场景图像，通过数据处理将采集到的场景图像显示在显示屏110上，用户在显示屏110上实时观察到的图像即为当前场景图像。可以理解，随着AR设备的位姿发生改变，采集到的当前场景图像随之改变，也即当前场景图像为实时图像。
标识对象指用于触发AR效果展示的标识物，也即上文所述的Marker，标识对象的具体呈现方式可以是例如二维码、预设图形等，本公开对此不作限制。
例如一个示例中，如图5所示，标识对象210为绘制于明信片200表面的卡通图形，该卡通图形可以基于例如汉明码技术进行绘制，从而AR设备在采集到包含该卡通图形的图像时，可以对卡通图形进行识别解析。对于汉明码绘制和解析的过程，本领域技术人员参照相关技术即可理解并充分实施，本公开对此不作限制。
例如另一个示例中，标识对象为绘制于物体表面的预设图形，AR设备在采集到包括该预设图形的场景图像时，可以通过图像检测技术对场景图像进行特征点提取和识别，从而可以从场景图像中识别到该预设图像。对于图像检测和预设图像的识别过程，本领域技术人员参照相关技术即可理解并充分实施，本公开对此不作限制。
可以理解，标识对象关联对应的虚拟对象，例如一个示例中，可以预先建立包括至少一个虚拟对象的模型数据库，模型数据库中包括虚拟对象以及与虚拟对象关联的标识对象，从而在AR设备由当前场景图像中识别到标识对象时，可以通过查找模型数据库，确定与该标识对象对应的虚拟对象。
虚拟对象表示为在AR设备的显示屏上经渲染呈现的虚拟形象，通过将虚拟对象与显示屏上显示的真实场景图像进行融合展示，可以呈现出虚拟对象与真实场景产生互动的展示效果。
本公开实施方式中，AR设备通过图像检测技术对采集到的当前场景图像进行检测识别，在从当前场景图像中识别到标识对象的情况下，即可确定与标识对象对应的虚拟对象。
AR设备在从当前场景图像中识别到标识对象的情况下，可以基于同步定位与建图(Simultaneous Localization And Mapping，SLAM)技术在AR设备上展示对应的虚拟对象。SLAM技术能够实现AR设备的空间定位和针对当前场景的地图构建，从而在AR设备显示的当前场景图像上渲染虚拟对象与现实场景叠加的展示效果，使虚拟对象的展示效果更加逼真。
具体来说，在基于SLAM进行AR效果展示时，需要将AR设备的位姿与虚拟对象的位姿映射到同一坐标系中，从而确定两者的相对位姿，然后基于相对位姿实时将虚拟对象映射到屏幕坐标系中，实时渲染AR效果。
在一些实施方式中，可以基于标识对象预先构建世界坐标系，然后根据虚拟对象在 世界坐标系中的实时位姿数据，以及AR设备在世界坐标系中的实时位姿数据，确定虚拟对象与AR设备之间的相对位姿数据，也即本公开所述的第一相对位姿数据，然后基于屏幕坐标系与世界坐标系的映射关系，根据第一相对位姿数据确定虚拟对象在屏幕坐标系中的展示位姿数据。本公开下文实施方式中具体进行说明，在此暂不详述。
S420、基于展示位姿数据，在AR设备展示虚拟对象。
具体而言，展示位姿数据包括在AR设备的屏幕坐标系中展示虚拟对象的相关数据，从而在S410得到展示位姿数据之后，即可根据该展示位姿数据将虚拟对象展示在屏幕中对应的位置，以使得用户可以在屏幕的对应位置观察到该虚拟对象。
S430、在虚拟对象的展示过程中，根据虚拟对象与标识对象在世界坐标系中的第二相对位姿数据，确定虚拟对象的目标展示数据。
具体而言，本公开实施方式中，在基于SLAM展示虚拟对象的过程中，同时利用图像检测技术，检测当前场景图像中的标识对象的位姿数据，从而确定当前场景图像中，标识对象与虚拟对象在世界坐标系中的相对位姿，也即本公开所述的第二相对位姿数据。
在一些实施方式中，可以针对虚拟对象与标识对象之间在世界坐标系中的不同的第二相对位姿数据，预先设置对应的展示数据，不同的展示数据对应不同的展示效果。从而在虚拟对象的展示过程中，实时检测虚拟对象与标识对象之间的相对位姿变化，也即第二相对位姿数据的变化，实时根据当前的第二相对位姿数据确定对应的目标展示数据，AR设备根据目标展示数据在显示屏上呈现对应的展示效果。
例如，第二相对位姿数据可以包括虚拟对象与标识对象在世界坐标系中的相对距离，并且，不同的相对距离对应有不同的展示数据。在一个示例中，可以随着相对距离增大，不同展示数据所对应的展示效果为：虚拟对象呈现透明度逐渐升高的展示效果。从而可以实时检测虚拟对象与标识对象的相对距离，根据当前的相对距离确定对应的透明度数据，也即目标展示数据。
当然，可以理解，第二相对位姿数据并不局限于虚拟对象与标识对象的相对距离，还可以是其他任何适于实施的相对位姿，例如相对角度、相对姿态等，本公开对此不作限制。而且，展示数据也不局限于虚拟对象的透明度数据，还可以是其他任何适于实施的展示数据，例如虚化效果、大小、亮度数据等，本公开对此不作限制。
S440、根据展示位姿数据和目标展示数据，在AR设备展示虚拟对象。
具体而言，目标展示数据指示针对虚拟对象渲染的展示效果，展示位姿数据表示虚拟对象在屏幕坐标系中的展示位姿。从而在确定目标展示数据之后，即可基于目标展示数据和展示位姿数据渲染展示虚拟对象。
在一个示例中，基于第二相对位姿确定目标展示数据为“以50％的透明度展示虚拟对象”，从而AR设备即可重新渲染虚拟对象，使得虚拟对象在AR设备的屏幕上以50％的透明度进行展示。
在另一个示例中，基于第二相对位姿确定目标展示数据为“以50％的尺寸展示虚拟对象”，从而AR设备即可重新渲染虚拟对象，使得虚拟对象在AR设备的屏幕上缩放至50％的尺寸进行展示。
当然可以理解，目标展示数据并不局限于上述示例，本公开对此不再枚举。
通过上述可知，本公开实施方式中，在AR效果展示过程中，可以根据标识对象与虚拟对象的相对位姿变化，实时展示不同的展示效果，从而增加AR场景的互动性，提高虚拟形象的展示效果。并且，基于虚拟对象与AR设备的第一相对位姿数据来展示虚 拟对象，使得虚拟对象的展示效果更加逼真，提高展示效果。本公开实施方式中，在AR效果展示时，AR设备需要从采集到的当前场景图像中识别出标识对象。如图6所示，在一些实施方式中，从当前场景图像中识别标识对象的过程包括：
S610、对当前场景图像进行特征点提取，得到当前场景图像包括的至少一个特征点对应的特征信息。
S620、将至少一个特征点对应的特征信息与预先存储的标识对象的特征点信息进行对比，识别当前场景图像中的标识对象。
具体而言，AR设备100可以通过摄像头120采集到当前场景图像，可以理解，当前场景图像可以是摄像头采集到的单帧图像，也可以是摄像头采集到的视频流中的多帧图像，本公开对此不作限制。
在对当前场景图像进行处理的过程中，可以通过图像检测算法，定位出当前场景图像中包括待测对象的检测区域，然后在检测区域中提取得到待测对象的一个或多个特征点。
以当前场景图像为图5为例，可以通过图像检测算法，首先确定明信片200区域为待测区域，然后对明信片200上包括的“小龙人”和“祥云”等待测对象进行特征点提取，得到每个待测对象的特征信息。
在得到各个待测对象的特征信息之后，可以将待测对象的特征信息与预先存储的标识对象的特征点信息进行对比，若两者相似度大于预设阈值，则表示待测对象即为标识对象；反之，若相似度不大于预设阈值，表示待测对象不是标识对象。
仍以图5为例，假设预先存储的标识对象的特征点信息对应“小龙人”卡通形象，从而将当前场景图像中提取到的小龙人的特征信息与预先存储的标识对象的特征点信息进行对比，两者相似度大于预设阈值，从而确定当前场景图像中识别到标识对象。
示例性地，本公开实施方式所述的特征点的信息可以包括特征点对应的纹理特征值、RGB特征值、灰度值等能够标识该特征点特征的信息，本公开对此不作限制。
在从当前场景图像中识别到标识对象时，即可进一步获取标识对象所对应的虚拟对象，以及该虚拟对象展示所需的展示位姿数据等。下面结合图7实施方式进行说明。
如图7所示，在一些实施方式中，本公开示例的展示方法，确定虚拟对象的展示位姿数据的过程包括：
S710、根据虚拟对象与AR设备的初始相对位姿，通过实时定位与建图(SLAM)确定第一相对位姿数据。
S720、根据第一相对位姿数据，以及世界坐标系与AR设备的屏幕坐标系的映射关系，确定虚拟对象在屏幕坐标系中的展示位姿数据。
具体而言，在基于SLAM进行虚拟对象展示时，需要将虚拟对象与AR设备对应到同一坐标系中，从而在两者位姿发生相对运动时，AR设备可以利用IMU传感器160实时解算确定两者的实时相对位姿，该坐标系即本公开所述的世界坐标系。
在一些实施方式中，可以基于图像检测技术预先对标识对象进行特征提取，得到标识对象的至少一个预设关键点，基于至少一个预设关键点建立世界坐标系。本公开下文中对此进行说明，在此暂不详述。
在对虚拟对象进行展示时，首先确定每个虚拟对象在世界坐标系中的预设初始位姿，也即第一初始位姿，第一初始位姿可以包括各个虚拟对象的初始位置和姿态。同时，可以在AR设备拍摄第一帧场景图像时，获取AR设备在世界坐标系中的初始位姿，也即 第二初始位姿，第二初始位姿同样可以包括AR设备的初始位置和姿态。在一些实施方式中，可以将AR设备100在拍摄第一帧场景图像时，IMU传感器160的位姿确定AR设备在世界坐标系中的初始位姿。
在同一世界坐标系下，针对每个虚拟对象，确定该虚拟对象的第一初始位姿和AR设备的第二初始位姿之后，即可根据AR设备的初始位姿与该虚拟对象的初始位姿得到虚拟对象与AR设备的初始相对位姿。例如，根据该虚拟对象与AR设备的位置差确定两者的相对位置，根据该虚拟对象与AR设备的初始姿态差确定两者的相对姿态。
在基于SLAM进行虚拟对象的AR效果展示时，随着AR设备的移动和/或转动，IMU传感器160可以检测到AR设备100的位姿(位置和姿态)发生变化。
例如，陀螺仪可以根据AR设备在三轴方向上与初始姿态的角度偏差，确定AR设备位姿变化后的姿态，加速度传感器可以检测到AR设备移动过程中的移动加速度，通过惯性导航解算方法得到移动后的位置，从而确定AR设备移动后的位姿数据。
在确定AR设备移动和/或转动后的位姿数据后，据此对前述的初始相对位姿进行更新，即可得到AR设备更新后的位姿与虚拟对象的位姿之间的相对位姿数据，也即第一相对位姿数据。
对于利用SLAM算法确定虚拟对象与AR设备之间相对位姿数据的过程，本领域技术人员根据相关技术可以理解并充分实施，本公开对此不再赘述。
在虚拟对象的AR效果展示时，为了将虚拟对象在AR设备100的显示屏110上显示，需要预先建立虚拟对象所在的世界坐标系与AR设备的屏幕坐标系之间的映射关系。例如一些实施方式中，可以将世界坐标系通过旋转和平移处理对应到AR设备的相机坐标系，然后从相机坐标系映射到AR设备的屏幕坐标系中。
对于世界坐标系与屏幕坐标系的对齐处理过程，本领域技术人员参照相关技术可以理解并充分实施，本公开对此不再赘述。
AR设备在实时确定自身与虚拟对象的第一相对位姿数据之后，即可将第一相对位姿数据根据上述映射关系，映射至AR设备的屏幕坐标系中，确定虚拟对象在屏幕坐标系中的展示位姿数据。然后AR设备根据展示位姿数据在AR设备的显示屏上渲染展示，使得用户可以观看到实时的虚拟对象的位姿变化。
在一个例子中，在用户佩戴例如图1所示的AR眼镜，逐渐移动远离初始位置时，基于图7所示的实施方式，即可在显示屏110上观看到虚拟对象逐渐缩小；当用户逐渐靠近初始位置时，即可在显示屏110上观看到虚拟对象逐渐变大。本公开实施方式展示效果符合人眼在真实场景中观看时的“远小近大”，因此虚拟对象的展示更加逼真。
在另一个例子中，在用户佩戴例如图1所示的AR眼镜，围绕虚拟对象转圈时，基于图7所示的实施方式，即可在显示屏110上观看到虚拟对象同步转动，例如由虚拟对象的正面逐渐转动至背面，仿佛虚拟对象固定在真实场景中，虚拟对象的展示更加逼真。
本领域技术人员对于上述实施方式的展示效果应当可以理解，并不局限于上述示例，本公开对此不再枚举。
通过上述可知，本公开实施方式中，在AR效果展示过程中，基于SLAM方式展示虚拟对象，使得虚拟对象的展示效果更加逼真，提高用户体验。
如图8所示，在一些实施方式中，本公开示例的展示方法，确定虚拟对象与AR设备的初始相对位姿的过程，包括：
S810、对当前场景图像进行图像检测，得到标识对象的预设关键点。
S820、基于标识对象的预设关键点，建立世界坐标系。
S830、根据虚拟对象在世界坐标系中的预设初始位姿，和AR设备在世界坐标系中的初始位姿，确定虚拟对象与AR设备的初始相对位姿。
具体而言，以当前场景图像为图5为例，标识对象210为绘制于明信片200表面的卡通形象，当AR设备通过前述图6实施方式，对当前场景图像检测识别到该当前场景图像包含标识对象时，可以基于标识对象的预设关键点，建立世界坐标系。
例如一个示例中，如图5所示，预设关键点可以是标识对象的中心点，将标识对象的中心点作为坐标系原点O，以平行于明信片200短边且穿过原点O的方向作为X轴，以平行于明信片200长边且穿过原点O的方向作为Y轴，以垂直于X轴和Y轴且穿过原点O的方向作为Z轴，从而建立世界坐标系O-XYZ。
可以理解，图5中建立世界坐标系的过程仅作为本公开实施方式的一个示例，在其他实施方式中，还可以依据其他预设关键点建立世界坐标系，例如可以明信片200左下角的顶点作为坐标原点O建立坐标系，本公开对此不作限制。
在基于SLAM进行虚拟对象展示时，可以预先设置虚拟对象在世界坐标系中的初始位姿，也即预设初始位姿，预设初始位姿可以包括虚拟对象的初始位置和姿态。例如图5示例中，虚拟对象的预设初始位姿包括：在世界坐标系的原点O处面朝-y方向站立，也即，预设初始位姿包括虚拟对象的位置(原点O)和姿态(面朝-y方向站立)。
同时，在建立世界坐标系之后，可以在AR设备拍摄第一帧场景图像时，获取AR设备在世界坐标系中的初始位姿，初始位姿同样可以包括AR设备的位置和姿态。在一些实施方式中，可以将，AR设备100在拍摄第一帧场景图像时，IMU传感器160的位姿确定为AR设备在世界坐标系中的初始位姿。
在同一世界坐标系下，确定虚拟对象的预设初始位姿和AR设备的初始位姿之后，即可根据两者得到虚拟对象与AR设备的初始相对位姿。例如，根据虚拟对象与AR设备的初始位置差确定两者的相对位置数据，根据虚拟对象与AR设备的初始姿态差确定两者的相对姿态数据，相对位置数据和相对姿态数据即组成虚拟对象与AR设备的初始相对位姿。
本公开实施方式中，在虚拟对象的展示过程中，不仅实时检测AR设备与虚拟对象的相对位姿变化，同时检测标识对象与虚拟对象的相对位姿变化，从而产生不同的展示效果，下面结合图9进行说明。
如图9所示，在一些实施方式中，本公开示例的展示方法，确定虚拟对象的目标展示数据的过程包括：
S910、在虚拟对象的展示过程中，获取虚拟对象在世界坐标系中的第一位姿数据，以及标识对象在世界坐标系中的第二位姿数据。
S920、根据第一位姿数据和第二位姿数据，确定第二相对位姿数据。
S930、根据第二相对位姿数据与预先设置的展示数据之间的对应关系，确定与第二相对位姿数据对应的目标展示数据。
在一个示例中，当前场景图像可如图5所示，AR设备在识别到当前场景图像(图5)中的标识对象210时，可以通过前述实施方式实现虚拟对象的展示，AR设备显示屏的展示效果如图10所示。
在虚拟对象的展示过程中，可以基于图像检测技术，同时检测标识对象的位姿，可以理解，在虚拟对象的展示过程中，标识对象的位姿可能会发生变化，例如用户移动明 信片200，使得AR设备检测到的当前场景图像中标识对象的位姿发生变化。
本公开实施方式中，可以获取虚拟对象在世界坐标系中的第一位姿数据，例如将前述的虚拟对象在世界坐标系中的预设初始位姿作为第一位姿数据。同时，AR设备可以检测标识对象在世界坐标系中的第二位姿数据，例如可以基于图像检测技术确定明信片200的检测框，根据明信片200位姿变化前后的检测框差异确定标识对象的第二位姿数据。
在得到第一位姿数据和第二位姿数据之后，即可根据第一位姿数据和第二位姿数据，确定虚拟对象与标识对象之间的相对位姿，也即本公开所述的第二相对位姿数据。
在一些实施方式中，可以预先基于第二相对位姿数据建立第二相对位姿数据与展示数据之间的对应关系，也即展示效果关系。
在一个例子中，第二相对位姿数据包括虚拟对象与标识对象在世界坐标系中的相对距离，对应的展示数据包括虚拟对象的透明度，从而预先建立的展示效果关系可以如下表1所示：
表1
相对距离 透明度
L0～L1 0％
L1～L2 25％
L2～L3 50％
在表1所示的展示效果关系中，在虚拟对象与标识对象的相对距离为L0～L1时，虚拟对象的展示数据为“透明度0％”；而在虚拟对象与标识对象的相对距离为L1～L2时，虚拟对象的展示数据为“透明度25％”；而在虚拟对象与标识对象的相对距离为L2～L3时，虚拟对象的展示数据为“透明度50％”。也即，虚拟对象与标识对象的相对距离越远，虚拟对象的展示数据所呈现的展示效果越透明。
在另一个例子中，第二相对位姿数据包括虚拟对象与标识对象在世界坐标系中的相对姿态(例如相对角度)，对应的展示数据包括虚拟对象的亮度，从而预先建立的展示效果关系可以如下表2所示：
表2
相对角度 亮度
α0～α1 100％
α1～α2 75％
α2～α3 50％
在表2所示的展示效果关系中，在虚拟对象与标识对象的相对角度为α0～α1时，虚拟对象的展示数据为“亮度100％”；而在虚拟对象与标识对象的相对角度为α1～α2时，虚拟对象的展示数据为“亮度75％”；而在虚拟对象与标识对象的相对角度为α2～α3时，虚拟对象的展示数据为“亮度50％”。也即，虚拟对象与标识对象的相对角度越大，虚拟对象的展示数据所呈现的亮度越低。
在又一个例子中，第二相对位姿数据同时包括虚拟对象与标识对象在世界坐标系中的相对距离和相对姿态，对应的展示数据同时包括虚拟对象的透明度和亮度，从而预先建立的展示效果关系可如下表3所示：
表3
Figure PCTCN2022134712-appb-000001
Figure PCTCN2022134712-appb-000002
也即，在表3所示的展示效果关系中，虚拟对象的透明度和亮度展示数据同时随第二相对位姿数据发生变化。
可以理解，上述仅示出了本公开实施方式中的一些示例，在其他实施方式中，预先建立的展示效果关系还可以包括其他的方式，例如展示数据还可以包括颜色、亮度、透明度、曝光度或展示角度中的一种或者多种数据，本公开对此不作限制。
在预先建立上述展示效果关系之后，在上述图9实施方式中，即可根据虚拟对象与标识对象的第二相对位姿数据，通过查找上述展示效果关系，找到当前第二相对位姿数据所对应的目标展示数据。
在一个例子中，展示效果关系如上表3所示，检测到的第二相对位姿数据包括：相对距离位于L1～L2，相对角度位于α2～α3。从而，根据表3可以确定目标展示数据为：透明度25％且亮度50％。
在确定目标展示数据之后，即可基于目标展示数据和前述确定的展示位姿数据实时渲染虚拟对象，以使用户可以在AR设备的显示屏上观看到目标展示数据所对应的虚拟对象的展示效果。
通过上述可知，本公开实施方式中，在AR效果展示过程中，可以根据标识对象与虚拟对象的相对位姿变化，实时展示不同的展示效果，从而增加AR场景的互动性，提高虚拟对象的展示效果。并且，在AR效果展示过程中，基于SLAM方式展示虚拟对象，使得虚拟对象的展示效果更加逼真，提高用户体验。
本公开实施方式提供了一种AR场景的展示装置，该装置可应用于AR设备。在一些实施方式中，AR设备可以是例如图1所示的AR眼镜。在另一些实施方式中，AR设备可以是例如图2a和图2b所示的智能手机。本公开对此不作限制。
如图11所示，在一些实施方式中，本公开示例的AR场景的展示装置，包括：
第一位姿确定模块10，被配置为响应于从当前场景图像中识别到标识对象，基于AR设备与标识对象对应的虚拟对象在世界坐标系中的第一相对位姿数据，确定虚拟对象的展示位姿数据；
第一展示模块20，被配置为基于展示位姿数据，在AR设备展示虚拟对象；
展示数据确定模块30，被配置为在虚拟对象的展示过程中，根据虚拟对象与标识对象在世界坐标系中的第二相对位姿数据，确定虚拟对象的目标展示数据；
第二展示模块40，被配置为根据展示位姿数据和目标展示数据，在AR设备展示虚拟对象。
通过上述可知，本公开实施方式中，在AR效果展示过程中，可以根据标识对象与虚拟对象的相对位姿变化，实时展示不同的展示效果，从而增加AR场景的互动性，提高虚拟形象的展示效果。并且，基于虚拟对象与AR设备的第一相对位姿数据来展示虚拟对象，使得虚拟对象的展示效果更加逼真，提高展示效果。在一些实施方式中，展示 数据确定模块30，被配置为：
在虚拟对象的展示过程中，获取虚拟对象在世界坐标系中的第一位姿数据，以及标识对象在世界坐标系中的第二位姿数据；
根据第一位姿数据和第二位姿数据，确定第二相对位姿数据；
根据第二相对位姿数据与预先设置的展示数据之间的对应关系，确定与第二相对位姿数据对应的目标展示数据。
在一些实施方式中，第二相对位姿数据包括虚拟对象与标识对象之间的相对位置和/或相对姿态。
在一些实施方式中，展示数据包括虚拟对象的颜色、亮度、透明度或展示角度中的一种或多种数据。
在一些实施方式中，第一位姿确定模块10，被配置为：
根据虚拟对象与AR设备的初始相对位姿，通过实时定位与建图确定第一相对位姿数据；世界坐标系为根据提取的标识对象的特征确定的，初始相对位姿为根据虚拟对象在世界坐标系的预设初始位姿和AR设备在世界坐标系的初始位姿确定；
根据第一相对位姿数据，以及世界坐标系与AR设备的屏幕坐标系的映射关系，确定虚拟对象在屏幕坐标系中的展示位姿数据。
在一些实施方式中，第一位姿确定模块10，被配置为：
对当前场景图像进行特征点提取，得到当前场景图像包括的至少一个特征点对应的特征信息；
将至少一个特征点对应的特征信息与预先存储的标识对象的特征点信息进行对比，识别当前场景图像中的标识对象。
通过上述可知，本公开实施方式中，在AR效果展示过程中，可以根据标识对象与虚拟对象的相对位姿变化，实时展示不同的展示效果，从而增加AR场景的互动性，提高虚拟形象的展示效果。并且，在AR效果展示过程中，基于SLAM方式展示虚拟对象，使得虚拟对象的展示效果更加逼真，提高用户体验。
本公开实施方式提供了一种AR设备，包括：
处理器；和
存储器，与处理器可通信连接，存储器存储有能够被处理器读取的计算机指令，计算机指令用于使处理器执行根据第一方面任一实施方式的方法。
本公开实施方式提供了一种存储介质，存储有计算机指令，计算机指令用于使计算机执行根据第一方面任一实施方式的方法。
具体而言，对于本公开示例的AR设备和存储介质相关实现方式，本领域技术人员参照前述实施方式可以理解并充分实施，本公开对此不再赘述。
本公开涉及增强现实领域，通过获取现实环境中的目标对象的图像信息，进而借助各类视觉相关算法实现对目标对象的相关特征、状态及属性进行检测或识别处理，从而得到与具体应用匹配的虚拟与现实相结合的AR效果。示例性的，目标对象可涉及与人体相关的脸部、肢体、手势、动作等，或者与物体相关的标识物、标志物，或者与场馆或场所相关的沙盘、展示区域或展示物品等。视觉相关算法可涉及视觉定位、SLAM、三维重建、图像注册、背景分割、对象的关键点提取及跟踪、对象的位姿或深度检测等。 具体应用不仅可以涉及跟真实场景或物品相关的导览、导航、讲解、重建、虚拟效果叠加展示等交互场景，还可以涉及与人相关的特效处理，比如妆容美化、肢体美化、特效展示、虚拟模型展示等交互场景。可通过卷积神经网络，实现对目标对象的相关特征、状态及属性进行检测或识别处理。上述卷积神经网络是基于深度学习框架进行模型训练而得到的网络模型。
上述实施方式仅仅是为清楚地说明所作的举例，而并非对实施方式的限定。对于所属领域的普通技术人员来说，在上述说明的基础上还可以做出其它不同形式的变化或变动。这里无需也无法对所有的实施方式予以穷举。而由此所引伸出的显而易见的变化或变动仍处于本公开创造的保护范围之中。

Claims (10)

  1. 一种增强现实AR场景的展示方法，其特征在于，所述方法包括：
    响应于从当前场景图像中识别到标识对象，基于AR设备与所述标识对象对应的虚拟对象在世界坐标系中的第一相对位姿数据，确定所述虚拟对象的展示位姿数据；
    基于所述展示位姿数据，在所述AR设备展示所述虚拟对象；
    在所述虚拟对象的展示过程中，根据所述虚拟对象与所述标识对象在所述世界坐标系中的第二相对位姿数据，确定所述虚拟对象的目标展示数据；
    根据所述展示位姿数据和所述目标展示数据，在所述AR设备展示所述虚拟对象。
  2. 根据权利要求1所述的展示方法，其特征在于，在所述虚拟对象的展示过程中，根据所述虚拟对象与所述标识对象在所述世界坐标系中的所述第二相对位姿数据，确定所述虚拟对象的所述目标展示数据，包括：
    在所述虚拟对象的展示过程中，获取所述虚拟对象在所述世界坐标系中的第一位姿数据，以及所述标识对象在所述世界坐标系中的第二位姿数据；
    根据所述第一位姿数据和所述第二位姿数据，确定所述第二相对位姿数据；
    根据所述第二相对位姿数据与预先设置的展示数据之间的对应关系，确定与所述第二相对位姿数据对应的所述目标展示数据。
  3. 根据权利要求2所述的展示方法，其特征在于，
    所述第二相对位姿数据包括所述虚拟对象与所述标识对象之间的相对位置和/或相对姿态。
  4. 根据权利要求2所述的展示方法，其特征在于，
    所述预先设置的展示数据包括所述虚拟对象的颜色、亮度、透明度或展示角度中的一种或多种数据。
  5. 根据权利要求1至4任一项所述的展示方法，其特征在于，基于所述AR设备与所述标识对象对应的所述虚拟对象在所述世界坐标系中的所述第一相对位姿数据，确定所述虚拟对象的所述展示位姿数据，包括：
    根据所述虚拟对象与所述AR设备的初始相对位姿，通过实时定位与建图确定所述第一相对位姿数据；所述世界坐标系根据所述标识对象的特征确定，所述初始相对位姿根据所述虚拟对象在世界坐标系的预设初始位姿和所述AR设备在世界坐标系的初始位姿确定；
    根据所述第一相对位姿数据，以及所述世界坐标系与所述AR设备的屏幕坐标系的映射关系，确定所述虚拟对象在所述屏幕坐标系中的所述展示位姿数据。
  6. 根据权利要求1至5任一项所述的展示方法，其特征在于，从所述当前场景图像中识别到所述标识对象，包括：
    对所述当前场景图像进行特征点提取，得到所述当前场景图像包括的至少一个特征点对应的特征信息；
    将所述至少一个特征点对应的特征信息与预先存储的所述标识对象的特征点信息进行对比，识别所述当前场景图像中的所述标识对象。
  7. 一种增强现实AR场景的展示装置，其特征在于，所述装置包括：
    第一位姿确定模块，被配置为响应于从当前场景图像中识别到标识对象，基于AR设备与所述标识对象对应的虚拟对象在世界坐标系中的第一相对位姿数据，确定所述虚拟对象的展示位姿数据；
    第一展示模块，被配置为基于所述展示位姿数据，在所述AR设备展示所述虚拟对象；
    展示数据确定模块，被配置为在所述虚拟对象的展示过程中，根据所述虚拟对象与所述标识对象在所述世界坐标系中的第二相对位姿数据，确定所述虚拟对象的目标展示数据；
    第二展示模块，被配置为根据所述展示位姿数据和所述目标展示数据，在所述AR设备展示所述虚拟对象。
  8. 根据权利要求7所述的展示装置，其特征在于，所述展示数据确定模块，被配置为：
    在所述虚拟对象的展示过程中，获取所述虚拟对象在所述世界坐标系中的第一位姿数据，以及所述标识对象在所述世界坐标系中的第二位姿数据；
    根据所述第一位姿数据和所述第二位姿数据，确定所述第二相对位姿数据；
    根据所述第二相对位姿数据与预先设置的展示数据之间的对应关系，确定与所述第二相对位姿数据对应的所述目标展示数据。
  9. 一种AR设备，其特征在于，包括：
    处理器；和
    存储器，与所述处理器可通信连接，所述存储器存储有能够被所述处理器读取的计算机指令，所述计算机指令用于使所述处理器执行根据权利要求1至6任一项所述的方法。
  10. 一种存储介质，其特征在于，存储有计算机指令，所述计算机指令用于使计算机执行根据权利要求1至6任一项所述的方法。
PCT/CN2022/134712 2021-12-31 2022-11-28 增强现实场景的展示 WO2023124693A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111667141.5 2021-12-31
CN202111667141.5A CN114332429A (zh) 2021-12-31 2021-12-31 增强现实ar场景的展示方法及装置

Publications (1)

Publication Number Publication Date
WO2023124693A1 true WO2023124693A1 (zh) 2023-07-06

Family

ID=81020457

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/134712 WO2023124693A1 (zh) 2021-12-31 2022-11-28 增强现实场景的展示

Country Status (2)

Country Link
CN (1) CN114332429A (zh)
WO (1) WO2023124693A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114332429A (zh) * 2021-12-31 2022-04-12 北京绵白糖智能科技有限公司 增强现实ar场景的展示方法及装置
CN114299262A (zh) * 2021-12-31 2022-04-08 北京绵白糖智能科技有限公司 增强现实ar场景的展示方法及装置
CN115619837B (zh) * 2022-12-20 2023-04-14 中科航迈数控软件(深圳)有限公司 一种ar图像生成方法及相关设备

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130215230A1 (en) * 2012-02-22 2013-08-22 Matt Miesnieks Augmented Reality System Using a Portable Device
CN112148197A (zh) * 2020-09-23 2020-12-29 北京市商汤科技开发有限公司 增强现实ar交互方法、装置、电子设备及存储介质
WO2021073268A1 (zh) * 2019-10-15 2021-04-22 北京市商汤科技开发有限公司 一种增强现实数据呈现方法、装置、电子设备及存储介质
CN114332429A (zh) * 2021-12-31 2022-04-12 北京绵白糖智能科技有限公司 增强现实ar场景的展示方法及装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130215230A1 (en) * 2012-02-22 2013-08-22 Matt Miesnieks Augmented Reality System Using a Portable Device
WO2021073268A1 (zh) * 2019-10-15 2021-04-22 北京市商汤科技开发有限公司 一种增强现实数据呈现方法、装置、电子设备及存储介质
CN112148197A (zh) * 2020-09-23 2020-12-29 北京市商汤科技开发有限公司 增强现实ar交互方法、装置、电子设备及存储介质
CN114332429A (zh) * 2021-12-31 2022-04-12 北京绵白糖智能科技有限公司 增强现实ar场景的展示方法及装置

Also Published As

Publication number Publication date
CN114332429A (zh) 2022-04-12

Similar Documents

Publication Publication Date Title
WO2023124693A1 (zh) 增强现实场景的展示
US11127210B2 (en) Touch and social cues as inputs into a computer
US10295826B2 (en) Shape recognition device, shape recognition program, and shape recognition method
US9779512B2 (en) Automatic generation of virtual materials from real-world materials
CN111598974B (zh) 用于呈现与真实对象相关的数字信息的方法和系统
US9224237B2 (en) Simulating three-dimensional views using planes of content
US10037614B2 (en) Minimizing variations in camera height to estimate distance to objects
US9123272B1 (en) Realistic image lighting and shading
US20130174213A1 (en) Implicit sharing and privacy control through physical behaviors using sensor-rich devices
US9979946B2 (en) I/O device, I/O program, and I/O method
EP3695381B1 (en) Floor detection in virtual and augmented reality devices using stereo images
US11069075B2 (en) Machine learning inference on gravity aligned imagery
US9906778B2 (en) Calibration device, calibration program, and calibration method
US20150378159A1 (en) Display control device, display control program, and display control method
US20150381970A1 (en) Input/output device, input/output program, and input/output method
US9449427B1 (en) Intensity modeling for rendering realistic images
WO2023124698A1 (zh) 增强现实场景的展示
WO2023124691A1 (zh) 增强现实场景的展示
EP3486875B1 (en) Apparatus and method for generating an augmented reality representation of an acquired image
US11763558B1 (en) Visualization of existing photo or video content
US11935199B2 (en) Augmented reality depth detection through object recognition
Malik et al. Private Try-On Using Marker Based Augmented Reality
GOUDA Augmented reality application for Polimi Leonardo campus using spatial indexing using R Tree

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22913951

Country of ref document: EP

Kind code of ref document: A1