WO2023071381A1 - Virtual-real fusion method and apparatus, and device, storage medium and program product - Google Patents

Abstract

Disclosed in the present disclosure are a virtual-real fusion method and apparatus, and a device, a storage medium and a program product. The virtual-real fusion method comprises: on the basis of a first positioning result of a target device before being calibrated, determining a first projection position of a virtual object in the current photographed picture of the target device, wherein the first projection position is determined to be a display position of the virtual object in the current photographed picture; and in response to a second positioning result of the target device after being calibrated having been acquired, and the virtual object meeting a first preset condition, continuing to determine the first projection position to be the display position of the virtual object in the current photographed picture.

Description

Virtual-real fusion method, device, equipment, storage medium and program product

Cross References to Related Applications

This disclosure is based on the Chinese patent application with the application number 202111272361.8, the application date is October 29, 2021, and the application name is "virtual-real fusion method, device, equipment and storage medium", and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into this disclosure in its entirety.

technical field

The present disclosure relates to the technical field of augmented reality, and in particular to a virtual-real fusion method, device, equipment, storage medium and program product.

Background technique

With the development of science and technology, augmented reality (Augmented Reality, AR) technology is more and more widely used. Augmented reality technology is a technology that combines real world and virtual world information. Display virtual visual information in real-world images through devices.

In the process of virtual-real fusion, due to the cumulative error in the positioning process of the device, it is necessary to update the position of the device at regular intervals in order to correct the cumulative error. Due to the existence of accumulated errors, after the location of the device is updated, the original virtual visual information will jump on the screen of the device, and the regular jump of this virtual visual information will bring a bad visual experience.

Contents of the invention

The present disclosure at least provides a virtual-real fusion method, device, equipment, storage medium and program product.

The present disclosure provides a virtual-real fusion method, including: based on the first positioning result of the target device before correction, determine the first projection position of the virtual object in the current shooting picture of the target device, wherein the first projection position is determined as the virtual object Display position in the current shooting frame; in response to obtaining the corrected second positioning result of the target device and the virtual object meets the first preset condition, continue to determine the first projection position as the virtual object’s position in the current shooting frame Show location.

Therefore, after obtaining the corrected second positioning result of the target device, in response to the fact that the virtual object satisfies the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting frame, which can reduce The visual impact caused by the jump in position.

The present disclosure provides a virtual-reality fusion device, including: a determining part configured to determine the first projection position of a virtual object on the current shooting picture of the target device based on the first positioning result of the target device before correction, wherein the first projection The position is determined as the display position of the virtual object in the current shooting frame; the fusion part is configured to respond to the acquisition of the corrected second positioning result of the target device, and the virtual object satisfies the first preset condition, continue the first projection position It is determined as the display position of the virtual object in the current shooting frame.

The present disclosure provides an electronic device, including a memory and a processor, and the processor is configured to execute program instructions stored in the memory, so as to realize the above virtual-real fusion method.

The present disclosure provides a computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the above virtual-real fusion method is realized.

The present disclosure provides a computer program product, including computer-readable codes. When the computer-readable codes are run in an electronic device, a processor in the electronic device is configured to implement the above virtual-real fusion method.

In the above solution, after obtaining the corrected second positioning result of the target device, in response to the fact that the virtual object satisfies the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting screen, which can reduce the risk of the virtual object due to The visual impact caused by the jump of the display position.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Description of drawings

The accompanying drawings here are incorporated into the description and constitute a part of the present description. These drawings show embodiments consistent with the present disclosure, and are used together with the description to explain the technical solution of the present disclosure.

FIG. 1 is a schematic flow diagram of an embodiment of the virtual-real fusion method of the present disclosure;

FIG. 2 is a schematic diagram of a part of the sub-flow of step S12 according to an embodiment of the virtual-real fusion method of the present disclosure;

Fig. 3 is another schematic flow chart of an embodiment of the virtual-real fusion method of the present disclosure;

Fig. 4 is a schematic structural diagram of an embodiment of the virtual reality fusion device of the present disclosure;

5 is a schematic structural diagram of an embodiment of the disclosed electronic device;

Fig. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present disclosure.

Detailed ways

The solutions of the embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings.

In the following description, for purposes of illustration rather than limitation, specific details, such as specific system architectures, interfaces, techniques, are set forth in order to provide a thorough understanding of the present disclosure.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship. In addition, "many" herein means two or more than two. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of an embodiment of a virtual-real fusion method of the present disclosure. The virtual-real fusion method may include the following steps S11 to S12:

Step S11: Based on the first positioning result of the target device before correction, determine the first projection position of the virtual object in the current shooting frame of the target device, wherein the first projection position is determined as the display position of the virtual object in the current shooting frame.

Wherein, based on the first positioning result of the target device before correction, the method of determining the first projection position of the virtual object in the current shooting frame of the target device may be determined according to the first positioning result of the target device and the original spatial position of the virtual object .

Wherein, the target device may be an execution device of the method provided by the embodiments of the present disclosure. In some disclosed embodiments, the target device may be a device with an augmented reality function, such as AR glasses, a mobile phone, and the like. In other disclosed embodiments, the target device may not be the execution device of the method provided by the embodiments of the present disclosure, that is, the target device may only perform shooting of the environment picture and display of the virtual-real fusion picture. In implementation, after the target device captures the environment picture, it transmits the environment picture to the execution device, and the execution device locates the target device to obtain the first positioning result of the target device before correction, and then based on the first positioning result and the virtual The original spatial position of the object, determine the first projection position of the virtual object on the current shooting screen of the target device, and then fuse the virtual object with the current shooting screen to obtain a fusion screen, and transmit the fusion screen to the target device so that the target device can The fusion screen is displayed. In the embodiment of the present disclosure, the target device is taken as an execution device of the method provided by the embodiment of the present disclosure as an example.

Virtual objects include visible display information or effects, such as visible text, graphics, models, animation images, etc.; virtual objects can also include invisible virtual objects used to block collisions, for example, virtual objects can be indicators, navigation, Points of interest, avatars, etc.

The first positioning result of the target device before correction may be the position or pose of the target device in the world coordinate system, and the pose includes position and pose. Wherein, the world coordinate system is aligned with the virtual space where the virtual object is located.

Step S12: In response to obtaining the corrected second positioning result of the target device and the virtual object meets the first preset condition, continue to determine the first projection position as the display position of the virtual object in the current shooting frame.

Among them, visual inertial positioning or other 6 degrees of freedom (6 Degree of Freedom, 6DoF) tracking algorithms are generally used in virtual-real fusion to obtain the positioning results of the target device in the world coordinate system. However, these positioning methods may have positioning errors, so it is inevitable that there will be certain cumulative errors in the continuous positioning process. Therefore, it is necessary to periodically correct the positioning results of the target device to reduce the cumulative error in the positioning process. The specific time for correcting the positioning result of the target device can be set by the user, or the setting of the target device when it leaves the factory can be adopted. For example, the first positioning result of the target device is corrected every 30 seconds (s).

Wherein, the first preset condition may be that the accuracy requirement set for the virtual object is smaller than the preset accuracy requirement. For example, the accuracy requirements for virtual objects of indication and navigation types are relatively loose compared to the accuracy requirements of virtual objects of label type. The virtual objects of the indication type and the navigation type can be virtual arrows, etc., and the virtual objects of the label type can be specific introductions (such as prices, product introductions, etc.) corresponding to each commodity in the store. The accuracy requirements of such indication-type and navigation-type virtual objects can be set to be smaller than the preset accuracy requirements. The accuracy requirement of the tag virtual object can be set to be greater than or equal to the preset accuracy requirement. Certainly, in some other embodiments, the first preset condition may be adjusted according to the allowable error requirement set by the virtual object.

When the corrected second positioning result of the target device is obtained and the virtual object satisfies the first preset condition, continue to determine the first projection position as the display position of the virtual object in the current shooting frame, which means Before and after device calibration, the display position of the virtual object in the current shooting screen does not change.

In the above solution, after obtaining the corrected second positioning result of the target device, in response to the fact that the virtual object satisfies the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting screen, which can reduce the risk of the virtual object due to The visual impact caused by the jump of the display position.

In some disclosed embodiments, after the display position is determined, the virtual-real fusion method further includes the following steps: displaying the current shooting frame, and displaying the virtual object at the first projection position of the current shooting frame.

In implementation, the shooting picture and the virtual object can be fused to obtain a fused picture. A possible fusion manner is to project the virtual object onto the first projection position of the current shooting frame. Then, displaying the fusion picture on the display interface of the target device can realize the display of the current shooting picture, and display the virtual object on the first projection position of the current shooting picture. That is to say, after determining the first projection position of the virtual object in the current shooting frame of the target device according to the first positioning result of the target device, and determining the first projection position as the display position of the virtual object in the current shooting frame , display the virtual object at the display position of the current shooting frame, and display the current shooting frame, and after step S12 is executed, continue to determine the first projection position as the display position of the virtual object in the current shooting frame, and continue to display the current shooting frame screen, and continue to display the virtual object at the display position in the current shooting screen.

Wherein, after determining the display position of the virtual object in the current shooting frame for the first time, the virtual object can be continuously displayed at the currently determined display position, and the display position of the virtual object can be updated during the display process. For example, before the positioning result of the target device is corrected, the display position of the virtual object in the current shooting frame is the first projection position; after the positioning result of the target device is corrected, and the virtual object satisfies the first preset condition In this case, the display position of the virtual object in the current shooting frame remains unchanged; in the case that the virtual object does not meet the first preset condition, the display position of the virtual object in the current shooting frame may change, then after the change The display position of continues to display the virtual object.

In some disclosed embodiments, the above step S11 includes: using the original spatial position of the virtual object, the projection parameters of the target device, and the second transformation parameters between the world coordinate system and the camera coordinate system of the target device to obtain the first projection position.

Wherein, the original space position of the virtual object is preset, which refers to the position of the virtual object in the virtual space. The second transformation parameter between the world coordinate system and the camera coordinate system of the target device is specifically a view matrix (View Matrix) of the target device. The second transformation parameter is a second transformation parameter transformed from the world coordinate system to the camera coordinate system of the target device. Wherein, the first positioning result of the target device may be the pose of the target device in the world coordinate system. Wherein, the pose of the target device includes the position and posture (or orientation) of the target device. In the embodiment of the present disclosure, the pose of the shooting component of the target device in the world coordinate system is equivalent to the pose of the target device in the world coordinate system. Of course, in some other disclosed embodiments, the pose of a preset component on the target device may be used as the pose of the target device. Then, the pose of the photographing component is determined according to the relative pose relationship between the photographing component on the target device and the preset position on the target device, and the pose of the preset component. Wherein, the position of the photographing component can be considered as the origin of the camera coordinate system. Specifically, according to the pose of the shooting component of the target device, reference may be made to general known technologies for a manner of determining the transformation relationship between the world coordinate system and the camera coordinate system of the target device.

The form of the projection parameter is a projection matrix, and the projection parameter is an internal parameter of the camera component, which is mainly constructed according to the projection principle, and is used to realize the transformation from the three-dimensional space coordinates to the two-dimensional image coordinates.

The second transformation parameter is used to determine the three-dimensional space coordinates of the virtual object in the camera coordinate system of the target device based on the original space position of the virtual object, and the projection parameter is used to determine the virtual object based on the three-dimensional space coordinates of the virtual object in the camera coordinate system. The first projection position of the object in the current shooting frame. By positioning and matching the target device with the preset map, a second positioning result of the target device in the preset map is obtained, so that the second positioning result is more accurate than the first positioning result before correction.

Wherein, before the above step S12, the following steps are also included:

The target device is positioned and matched with the preset map to obtain a second positioning result of the target device in the preset map. The preset map may be a high-precision three-dimensional map, such as a dense three-dimensional map. Positioning and matching the target device with the preset map can also be referred to as obtaining a more accurate pose of the target device in the preset map. A manner of positioning and matching the target device with the preset map may be to obtain a second positioning result of the target device in the preset map according to an accurate positioning component in the target device. The precise positioning component may be a Global Positioning System (Global Positioning System, GPS). Of course, a more accurate positioning result of the target device in the preset map may also be obtained by using a visual-inertial positioning method. The feature points of the currently photographed picture are extracted and matched with the feature points in the preset map, and a second positioning result is obtained based on the matching results. In other disclosed embodiments, the current shooting picture can be sent to the cloud, so that the cloud can extract the feature points of the current shooting picture, and match the extracted feature points with the feature points in the preset map to obtain the target device. The second positioning result, and then receiving the second positioning result sent by the cloud.

In other disclosed embodiments, before the above step S12, the following steps may also be included:

Detecting whether there is a preset change between the corrected second positioning result and the first positioning result of the target device. In response to a preset change between the second positioning result and the first positioning result, step S12 is executed.

Wherein, the preset change may be that the error between the first positioning result and the second positioning result is greater than or equal to the preset error. The preset error can be determined according to the user's requirements for the accuracy of virtual-real fusion. In the case of higher requirements on accuracy, the preset error can be set smaller and close to 0; otherwise, the preset error can be set larger.

In some disclosed embodiments, the first preset condition includes at least one of the following: the virtual object meets the second preset condition, and the position deviation between the first modified spatial position and the original spatial position of the virtual object meets the preset position requirement. In some application scenarios, the first preset condition includes that the virtual object meets the second preset condition. In some application scenarios, the first preset condition includes that the position deviation between the first changed space position of the virtual object and the original space position meets the preset position requirements. In some application scenarios, the first preset condition includes that the virtual object meets the second preset condition, and the position deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirement.

Wherein, the second preset condition includes that the virtual object belongs to the first type and/or the first change space position of the virtual object belongs to a reasonable position. The first modified spatial position is determined based on the second positioning result of the target device, and the first modified spatial position can be projected to the spatial position of the first projected position. That is to say, when the target device is located at the second positioning result, the virtual object in the first change space can be projected to the first projection position of the current shooting frame. Before step S12 is executed, the way to obtain the first changed space position of the virtual object can be:

Using the first projected position, the projected parameters of the target device, and the first transformation parameters between the world coordinate system and the camera coordinate system of the target device, a first changed space position on the world coordinate system is obtained. Wherein, the first transformation parameter is obtained based on the second positioning result of the target device. For a manner of obtaining the first transformation parameter, reference may be made to the foregoing manner of obtaining the second transformation parameter.

Through the projection parameters, the virtual object can be back-projected from the first projection position in the current shooting frame to the camera coordinate system, and the three-dimensional coordinates of the virtual object in the camera coordinate system can be obtained, and then based on the first transformation parameter, the virtual object can be transformed from the camera coordinate system Convert to the world coordinate system to obtain the first changed space position of the virtual object on the world coordinate system. The first changed space position of the virtual object is determined by the second positioning result of the target device and the first projected position of the virtual object, so that the projection position of the virtual object at the first changed space position corresponding to the current shooting frame can be kept unchanged.

In some disclosed embodiments, the second preset condition includes that the virtual item belongs to the first type. Optionally, an instruction from the user to classify the virtual objects is received, and the virtual objects are classified into several types, for example, into three types. Different types of virtual objects have different fusion accuracy. The division principle of the virtual object may be determined according to the balance relationship between the visual impact brought by the jump of the virtual object and the impact brought by the virtual object projection error. For example, when the projection error of a virtual object is relatively large, and the impact of the projection error is negligible when the virtual object does not jump, the virtual object can be classified into the first type, for example, for some large-scale navigation Arrows, the display of information panels, etc., actually deviate from the position of a few meters, and will not affect the navigation effect. In the case where the visual impact caused by the jump of the virtual object is less than the impact caused by the projection error, the virtual object may be classified into the second type, such as an information sign of a product. In the case where the visual impact caused by the jump of the virtual object is greater than the impact caused by the projection error, the virtual object may be classified into a third type, such as a virtual object used for occlusion.

In some disclosed embodiments, the second preset condition includes that the first change space position of the virtual object belongs to a reasonable position. Whether the first changed space position of the virtual object is a reasonable position can be specifically determined according to the semantic information at the first changed space position in the preset map. Among them, semantic information such as dangerous areas and heights in high-precision maps. Optionally, the reasonable location includes at least one of a non-dangerous area and a preset height range. A possible implementation manner is that dangerous areas can be pre-marked on a preset map, and areas other than the dangerous areas can be considered as non-dangerous areas. For example, in a city navigation scene, the virtual object is a navigation arrow, and the dangerous area may be a lake beside the road. The preset height range can be set according to the demand for projection accuracy. In order to better understand the preset height range, the following specific application examples are shown below. Assume that the preset map is a map of a certain room, the virtual object is a vase, and the original spatial position of the virtual object is on the table surface of the room. If the first change space position of the virtual object is below the floor of the room, this is obviously illogical, at least the base of the vase should be above the floor of the room, that is, the height of the base of the vase should be higher than the height of the ground. The preset height range is the maximum height and/or the minimum height allowed in the vertical direction where the first change space position is located in the preset map. By combining the semantic information of the preset map, it is determined whether the first change space position is a reasonable position, so that the determination of the reasonable position is more accurate, and the danger or other problems caused by not using the precise position are reduced.

In some disclosed embodiments, the second preset condition includes that the virtual object belongs to the first type, and the first change space position of the virtual object belongs to a reasonable position. That is, in response to the fact that the virtual object belongs to the first type and the first changed spatial position of the virtual object belongs to a reasonable position, the virtual object is kept displayed at the first projection position of the current shooting frame.

In some disclosed embodiments, the first preset condition includes that the position deviation between the first changed space position of the virtual object and the original space position meets the preset position requirement. The positional deviation between the first modified spatial position and the original spatial position may include a deviation in the height direction, or may not include a deviation in the height direction. For example, in some application scenarios, the position deviation may only include a deviation between two positions on a plane perpendicular to the height direction. Specifically, it may be a distance deviation on this plane. Wherein, the preset position requirement may include that the position deviation is smaller than a preset threshold. That is, the deviation between the original spatial position and the first changed spatial position is relatively small, and the impact of the position deviation is less than the impact of the jump of the virtual object in the current shooting frame. Preset thresholds can be user-defined, or default to the original settings of the target device. If the position deviation between the first changed space position of the virtual object and the original space position is less than the preset threshold, the deviation between the display positions of the two positions on the current shooting screen will not be large, so the display of the virtual object is maintained The position does not change, and compared with the display position change of the virtual object, the visual impact caused by the former is smaller.

Optionally, according to whether the virtual object is visible or not during the display change process, the preset position requirement may also include that the virtual object is visible during the display change process. That is, the display change process of the virtual object is visible and the position deviation is smaller than the preset threshold. Wherein, the display change process of the virtual object is a process in which the virtual object is displayed from the first position to the target projection position. Specifically, the joint bounding box formed by the virtual object at the original space position and the first changed space position is acquired. That is, the joint bounding box is a three-dimensional bounding box, which can be regarded as a cuboid, which wraps the virtual object at the original spatial position and the first changed position. The process of the virtual object jumping in the current shooting frame can be specifically considered as the projection of the virtual object on the current display interface during the process of moving from the first changed position to the original space position. During the current shooting process, it is mainly reflected in the interpolation process during the movement of the virtual object from the first projection position to the target projection position. The target projection position refers to the projection position of the virtual object in the current shooting frame obtained based on the original spatial position of the virtual object, the first transformation parameter and the projection parameter. Wherein, the invisible change process means that the interpolation process is invisible in the current shooting frame, and the visible change process means that the interpolation process is visible in the current shooting frame. Whether the interpolation process is visible can be determined according to whether there is a target projection position. In some implementation manners, if there is no target projection position corresponding to the virtual object in the current shooting frame, the interpolation process is considered invisible. In some implementations, when the target projection position is not located in the current shooting frame, that is, the target device cannot observe the virtual object at all based on the corrected position, the virtual object cannot Two transformation parameters and projection parameters, to obtain the projection position of the virtual object in the current shooting frame. At this time, the projection of the virtual object during movement in the joint bounding box cannot be reflected in the current shooting frame. Alternatively, whether the interpolation process is visible may be determined according to whether the interpolation process is complete. In some implementation manners, if the interpolation process is complete, the interpolation process is considered visible; in other implementation manners, if the interpolation process is incomplete, the interpolation process is considered invisible. For example, during the movement of the virtual object in the joint bounding box, the target device at the middle position cannot be observed, and thus cannot be displayed in the current shooting frame, so the interpolation at this position is considered invisible.

In some application scenarios, when the display change process of the virtual object is invisible, it can be considered that the position deviation between the first changed spatial position of the virtual object and the original spatial position meets the preset position requirements. In other application scenarios, because the display change process of the virtual object is invisible, even if the virtual object jumps, the impact on the visual perception is small, and the projection position of the virtual object can be corrected to reduce the position deviation. coming impact.

In some disclosed embodiments, the first preset condition includes that the virtual object meets the second preset condition, and the position deviation between the first changed spatial position and the original spatial position of the virtual object meets the preset position requirement. Based on FIG. 1 , FIG. 2 is a schematic diagram showing part of the sub-flow of step S12 in an embodiment of the virtual-real fusion method of the present disclosure. As shown in Figure 2, above-mentioned step S12 comprises the following steps:

Step S121: Determine whether the virtual object belongs to the first type.

In the implementation, firstly, the type of the virtual object is determined according to the preset parameter information. Wherein, the preset parameter information includes at least one type information corresponding to the virtual object. The parameter information of the preset setting can be customized by the user.

If the virtual item does not belong to the first type, execute step S122, and if the virtual item does not belong to the first type, execute step S123.

Step S122: Determine that the virtual item does not meet the first preset condition.

Step S123: Determine whether the positional deviation between the first modified spatial position of the virtual object and the original spatial position meets the preset position requirements.

Please refer to the above description for specific requirements on the preset location. In the case that the position deviation between the first modified spatial position of the virtual object and the original spatial position meets the requirement of the preset position, step S124 is executed; otherwise, step S122 is executed.

Step S124: Determine whether the first changed space position of the virtual object belongs to a reasonable position.

Wherein, for the definition of a reasonable location, please refer to the above description. If the first changed space position is within the range of the reasonable position, it is determined that the first changed space position of the virtual object belongs to the reasonable position. When the first change space of the virtual object does not belong to the reasonable position, it is determined that the first change space of the virtual thing belongs to the reasonable position. When the first space of the virtual object does not belong to a reasonable position, step S122 is executed. When the first space of the virtual object belongs to a reasonable position, step S125 is executed.

Step S125: Determine that the virtual object satisfies the first preset condition.

Wherein, the execution order of the above-mentioned step S123 and step S124 can be exchanged, and it is not strictly executed according to the above-mentioned step flow.

Through the type of virtual object, the first changed space position of the virtual object is in a reasonable position, and the position deviation between the first changed space position of the virtual object and the original space position meets the preset position requirements, compared with the display position of the virtual object In terms of jumping, the visual impact caused by a certain deviation in the display position is smaller.

In some disclosed embodiments, the virtual-real fusion method may also include the following steps:

In response to obtaining the corrected second positioning result of the target device and the virtual object does not meet the first preset condition, the first correction strategy or the second correction strategy is used to change the display position of the virtual object from the first projection position to the target device. projection location. Wherein, the target projection position may be obtained from the original space position of the virtual object and the above-mentioned first transformation parameters and projection parameters, and may also be determined based on setting parameters of the target device. Wherein, the determination based on the setting parameters of the target device may be based on the rationality determination of the first changed space position of the virtual object. By changing the display position of the virtual object when the virtual object does not satisfy the first preset condition, the bad visual experience caused by the deviation of the projection position of the virtual object can be reduced.

Specifically, before the first correction strategy or the second correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position, the target projection position is acquired first. Among them, there are many ways to obtain the target projection position:

The first method is to determine the second projection position of the virtual object in the current shooting frame as the target projection position based on the second positioning result and the original spatial position. Wherein, based on the second positioning result and the original space position, the specific process of determining the second projection position of the virtual object in the current shooting picture is as follows: based on the second positioning result of the target device, the distance between the camera coordinate system of the target device and the world coordinate system is obtained. between the first transformation parameters. For a specific manner of obtaining the first transformation parameter, please refer to the above description. Then use the first transformation parameter to convert the original space position of the virtual object to the camera coordinate system of the target device, and then use the projection parameter to project the virtual object in the camera coordinate system to the current shooting frame, and obtain the first position of the virtual object in the current shooting frame Two projection positions. The second projection position of the virtual object in the current shooting frame is used as the target projection position. Wherein, the projection position separated from the second projection position by a preset distance may also be used as the target projection position. For example, a projection position that is 2 pixels away from the second projection position may also be used as a target projection position.

The second method is to adjust the first changed space position to a second changed space position belonging to a reasonable position when the first changed space position does not belong to a reasonable position. Based on the second positioning result and the second modified spatial position, a third projection position of the virtual object on the current shooting frame is determined as a target projection position. A manner of adjusting the first changed space position to a second changed space position belonging to a reasonable position may be to acquire a relative position between the first changed space and the reasonable position, and adjust the first changed space position based on the relative position. Following the above example, the bottom of the virtual vase should be placed above the ground, and when the vase is in the first change space position, the bottom of the vase is 0.5 meters (m) below the ground, which is the relative distance between the vase and the reasonable position. The position is 0.5m, so move the vase up at least 0.5m from the first altered position so that the bottom of the vase is in contact with or above the ground. Thus, the second changed space position where the virtual object belongs to a reasonable position is obtained. Then, based on the second positioning result of the target device, the first transformation parameter between the camera coordinate system of the target device and the world coordinate system is obtained. For a specific manner of obtaining the first transformation parameter, please refer to the above description. Then use the first transformation parameter to convert the second changed space position of the virtual object to the camera coordinate system of the target device, and then use the projection parameter to project the virtual object in the camera coordinate system to the current shooting frame, and obtain the virtual object in the current shooting frame The second projection position of . The second projection position of the virtual object in the current shooting frame is used as the target projection position. In this way, the alignment effect between the virtual object and the shooting screen can be adjusted. By determining the target projection position based on the second positioning result and the original space position, or adjusting the second change space position belonging to a reasonable position according to the first change space position and determining the target projection position based on the second change space position, the virtual object After the display position is corrected, the adverse effects caused by projection errors are well reduced.

In some disclosed embodiments, the first correction strategy is to directly render the virtual object to the target projection position in the next display rendering process. That is, the target projection position of the virtual object on the target device is determined directly according to the position of the virtual object in the original space position, and then the virtual object is displayed on the target projection position of the target device. The second correction strategy is to move the virtual object from the first projection position to the target projection position at a preset speed. In some implementations, the first correction strategy can be considered as immediately changing the display position of the virtual object from the first projection position to the target projection position, and not displaying the interpolation process between the virtual object changing from the first projection to the target projection position , and the second correction strategy can be considered as gradually moving the display position of the virtual object from the first projection position to the target projection position. In some implementations, the second correction strategy is to move the virtual object from the first changed space position to the original space position or the corresponding second changed space position at a constant speed, and determine the corresponding projected position according to the current position of the virtual object in turn , and display at the corresponding projection position, so as to move the virtual object from the first projection position to the target projection position at a preset speed for display. Similarly, the first correction strategy is to directly switch the first changed space position of the virtual object to the original space position or the second changed space position, so as to directly render the virtual object to the target projection position in the next display rendering process. By setting different correction strategies, the correction process is more flexible.

Specifically, in response to the virtual object belonging to the second type, a first correction strategy is used to change the display position of the virtual object from the first projection position to the target projection position. Or in response to the fact that the virtual object belongs to the third type, the second correction strategy is used to change the display position of the virtual object from the second projection position to the target projection position. In response to the fact that the first changed space position does not belong to a reasonable position, a first correction strategy or a second correction strategy is selected to change the display position of the virtual object from the first projection position to the target projection position. Specifically, even if the virtual object does not have high requirements for projection accuracy and belongs to the first type, if the first change space position of the virtual object does not belong to a reasonable position, you can also choose the first correction strategy or the second correction strategy to make the virtual object The display position of the object is changed from the first projection position to the target projection position. As mentioned above, the classification of the types of virtual objects may be determined according to the requirements of the corresponding projection accuracy of the virtual objects. At this time, the second type has higher requirements on the projection position accuracy than the third type, and the third type has higher requirements on the projection position accuracy than the first type has on the projection position accuracy. That is, for the second type, if the correction strategy is not implemented, it will lead to worse user experience, so the correction strategy needs to be implemented immediately. For the first type, the projection deviation of the virtual object in the current shooting screen has little effect on the fusion of virtual and real. Under certain circumstances, the implementation of the deviation correction strategy will cause jumps and affect the experience. It is not necessary to correct the deviation. The case of the first preset condition. For the third type, there is little difference between the impact of the projection deviation of the virtual object in the current shooting frame and the impact of the jump caused by the implementation of the deviation correction strategy, so the deviation correction strategy can be implemented for it. By implementing different correction strategies for different types of virtual objects, the flexibility of the correction strategy manner is improved.

In some disclosed embodiments, when the preset position requires that the display change process including the virtual object be visible, and the position deviation is less than a preset threshold, in response to the acquired second corrected positioning result of the target device, and the virtual object If the first preset condition is not met, the way to change the display position of the virtual object from the first projection position to the target projection position by using the first correction strategy or the second correction strategy may be:

In response to the fact that the virtual object meets the second preset condition, the display change process of the virtual object is visible, and the position deviation is not less than a preset threshold, a first correction strategy is used to change the display position of the virtual object from the first projection position to the target projection position. Wherein, the position deviation here is the position deviation between the original space position and the first changed space position of the above-mentioned virtual object. That is, the virtual object belongs to the first type and/or the first changed space position of the virtual object belongs to a reasonable position, the display change process of the virtual object is visible, and the position deviation between the original space position of the virtual object and the first changed space position is not less than If the threshold is preset, the first correction strategy is used to change the display position of the virtual object.

In response to the fact that the virtual object meets the second preset condition and the display changing process of the virtual object is invisible, a second correction strategy is used to change the display position of the virtual object from the first projection position to the target projection position. Wherein, the position deviation here is the position deviation between the original space position and the first changed space position of the above-mentioned virtual object. That is, if the virtual object belongs to the first type and/or the first change space position of the virtual object belongs to a reasonable position, and the display change process of the virtual object is invisible, then the second correction strategy is adopted to change the display position of the virtual object. In some other disclosed embodiments, in response to the fact that the virtual item meets the second preset condition and the display changing process of the virtual item is invisible, the first correction strategy may also be selected to change the display position of the virtual item. By correcting the display position of the virtual object on the current shooting interface that satisfies the second preset condition but does not meet the preset position requirement, it is possible to reduce adverse effects caused by projection deviation of the virtual object on the current shooting interface.

In some disclosed embodiments, in response to the virtual object meeting the first preset condition, the method of keeping the virtual object displayed at the first projection position of the current shooting screen may be to keep the screen displayed by the target device unchanged, and another method may be The first changed space position of the virtual object is obtained according to the above method, and then based on the first changed space position of the virtual object and the first transformation parameters and projection parameters between the camera coordinate system of the target device and the world coordinate system, the virtual object is obtained at The projection position on the current shooting screen is used as the target projection position, so as to keep the virtual object displayed on the first projection position of the current shooting screen.

To better understand the above scenario, see the example below. Please refer to FIG. 3 at the same time. FIG. 3 is another schematic flowchart of an embodiment of the virtual-real fusion method of the present disclosure. The virtual-real fusion method provided by the embodiment of the present disclosure includes the following steps:

Step S21: Based on the first positioning result of the target device before being corrected, determine the first projection position of the virtual object on the current shooting frame of the target device.

Step S22: Display the current shooting frame, and display the virtual object on the first projection position of the current shooting frame.

Step S23: Obtain a corrected second positioning result of the target device.

Wherein, the manner of obtaining the corrected second positioning result of the target device is as above. Wherein, the second positioning result can be used to determine the target projection position of the virtual object.

Step S24: Determine whether the virtual object satisfies the first preset condition.

Wherein, the first preset condition is as above. If the virtual object satisfies the first preset condition, step S25 is executed; otherwise, one of the corresponding steps S26, S27 or S28 is executed.

Step S25: In response to the virtual object meeting the first preset condition, keep displaying the virtual object at the first projection position of the current shooting frame.

Wherein, the manner of maintaining and displaying the virtual object at the first projection position of the current shooting frame is as described above.

Step S26: In response to the fact that the virtual object belongs to the second type, a first correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position.

Wherein, the way of using the first correction strategy to change the display position of the virtual object from the first projection position to the target projection position is as described above.

Step S27: In response to the fact that the virtual object belongs to the third type, a second correction strategy is adopted to change the display position of the virtual object from the first projection position to the target projection position.

Wherein, the method of changing the display position of the virtual object from the first projection position to the target projection position by using the second correction strategy is as described above.

Step S28: In response to the fact that the first changed spatial position does not belong to a reasonable position, select the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position.

Specifically, in response to the fact that the virtual object belongs to the first type and the virtual object does not meet the first preset condition, the first correction strategy or the second correction strategy is selected to change the display position of the virtual object from the first projection position to the target projection position .

In the above solution, after obtaining the corrected second positioning result of the target device, in response to the fact that the virtual object satisfies the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting screen, which can reduce the risk of the virtual object due to The visual impact caused by the jump of the display position.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

The embodiment of the present disclosure is based on the virtual object type, the projection of the virtual object, and the back projection after positioning, to ensure that even if the virtual object has cumulative errors in six-degree-of-freedom tracking, according to the virtual object at the precise position and the virtual object at the inaccurate position, respectively Different strategies are used to prevent jumps to improve the user experience of virtual reality fusion.

The embodiments of the present disclosure are at least used in the following scenarios: for example, in airports, factories, venues, shopping malls, hospitals, and garages, augmented reality can realize the fusion effect of virtual reality in large-scale spaces. Through the virtual fusion method provided by the embodiments of the present disclosure, positioning Before and after, the position of the virtual object is recalculated to ensure that the projection position of the virtual object remains unchanged, avoiding jumps before and after positioning, and the user experience is more natural.

The execution subject of the virtual-real fusion method may be a virtual-real fusion device. For example, the virtual-real fusion method may be executed by a terminal device or a server or other processing equipment, wherein the terminal device may be an augmented reality device (AR device), such as an augmented reality or virtual reality Glasses, helmets and other products User Equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices as well as self-driving cars, robots, and more. In some possible implementation manners, the virtual-real fusion method may be implemented by calling a computer-readable instruction stored in a memory by a processor.

Please refer to FIG. 4 . FIG. 4 is a schematic structural diagram of an embodiment of a virtual reality fusion device of the present disclosure. The virtual reality fusion device 40 includes a determination part 41 and a fusion part 42 . The determining part 41 is configured to determine a first projection position of the virtual object in the current shooting frame of the target device based on the first positioning result of the target device before correction, wherein the first projection position is determined as the position of the virtual object in the current shooting frame display position; the fusion part 42 is configured to continue to determine the first projection position as the position of the virtual object in the current shooting frame in response to obtaining the corrected second positioning result of the target device and the virtual object meets the first preset condition Show location.

In the above solution, after obtaining the corrected second positioning result of the target device, in response to the fact that the virtual object satisfies the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting screen, which can reduce the risk of the virtual object due to The visual impact caused by the jump of the display position.

In some disclosed embodiments, after the display position is determined, the fusion part 42 is further configured to: display the current shooting frame, and display the virtual object at the display position in the current shooting frame.

In the above solution, after the target device is corrected, the virtual object satisfies the first preset condition, and the virtual object is kept displayed at the display position of the current shooting picture, which can reduce the visual impact caused by the jump of the display position.

In some disclosed embodiments, the first preset condition includes at least one of the following: the virtual object meets the second preset condition, the positional deviation between the first changed space position of the virtual object and the original space position meets the preset position requirement, and The second preset condition includes that the virtual object belongs to the first type and/or the first changed spatial position of the virtual object belongs to a reasonable position; wherein, the first changed spatial position is a spatial position determined based on the second positioning result of the target device, and the first Changing the spatial position projected from the spatial position to the first projected position.

The above scheme, through the type of virtual object, the first changed spatial position of the virtual object is in a reasonable position, and the position deviation between the first changed spatial position of the virtual object and the original spatial position meets the preset position requirements, compared with the virtual object’s As far as the display position jumps, the visual impact caused by a certain deviation in the display position is smaller.

In some disclosed embodiments, the preset position requirement includes a position deviation smaller than a preset threshold.

In the above solution, if the position deviation between the first changed space position of the virtual object and the original space position is less than the preset threshold, the deviation between the display positions of the two positions on the current shooting screen will not be large, so the virtual object remains virtual. The display position of the object does not change, compared with the change of the display position of the virtual object, the visual impact caused by the former is smaller.

Wherein, the preset position requirement also includes that the display change process of the virtual object is visible, wherein the display change process is a process in which the display of the virtual object changes from the first projection position to the target projection position.

Therefore, by keeping the display position of the virtual object from jumping when the display change process of the virtual object is visible and the position deviation is smaller than the preset threshold, compared with the jumping of the display position of the virtual object, the former brings Comes with less visual impact.

In some disclosed embodiments, the first preset condition includes that the virtual object meets the second preset condition, and the second preset condition includes that the first change space position of the virtual object belongs to a reasonable location; wherein, the reasonable location includes a non-dangerous area and belongs to At least one of the preset height ranges; and/or, before the first projection position is continuously determined as the display position of the virtual object in the current shooting frame in response to the virtual object meeting the first preset condition, the fusion part 42 It is further configured to: determine whether the first change space position belongs to a reasonable position based on the semantic information of the preset map.

In the above solution, by combining the semantic information of the preset map, it is determined whether the first changed spatial position belongs to a reasonable position, so that the determination of the reasonable position is more accurate.

In some disclosed embodiments, the fusion part 42 is further configured to: in response to obtaining the corrected second positioning result of the target device and the virtual object does not satisfy the first preset condition, adopt the first correction strategy or the second correction strategy to The display position of the virtual object is changed from the first projection position to the target projection position.

In the above solution, by changing the display position of the virtual object when the virtual object does not satisfy the first preset condition, the bad visual experience caused by the deviation of the projection position of the virtual object can be reduced.

In some disclosed embodiments, in response to obtaining the corrected second positioning result of the target device and the virtual object does not meet the first preset condition, the fusion part 42 adopts the first correction strategy or the second correction strategy to change the display of the virtual object to Changing the position from the first projection position to the target projection position includes: in response to the virtual object belonging to the second type, using the first correction strategy to change the display position of the virtual object from the first projection position to the target projection position; in response to the virtual object belonging to The third type is to change the display position of the virtual object from the first projection position to the target projection position by using the second correction strategy; The display position of the object is changed from the first projection position to the target projection position.

The above solution improves the flexibility of the correction strategy by implementing different correction strategies for different types of virtual objects.

In some disclosed embodiments, the preset position requirements include that the display change process of the virtual object is visible, and the position deviation is less than a preset threshold, wherein the display change process is a process in which the display of the virtual object changes from the first projection position to the target projection position; In response to obtaining the corrected second positioning result of the target device and the virtual object does not satisfy the first preset condition, the fusion part 42 uses the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to Changing the target projection position includes: adopting the first correction strategy to change the display position of the virtual object from the first to The projection position is changed at the target projection position; in response to the fact that the virtual object meets the second preset condition and the display change process of the virtual object is not visible, a second correction strategy is used to change the display position of the virtual object from the first projection position to the target projection position .

In the above solution, by correcting the display position of the virtual object on the current shooting interface that meets the second preset condition but not meeting the preset position requirements, it is possible to reduce the error caused by the projection deviation of the virtual object on the current shooting interface. Negative Effects.

In some disclosed embodiments, before adopting the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to the target projection position, the fusion part 42 further includes: based on the second positioning result and the original spatial position , to determine the second projection position of the virtual object in the current shooting picture as the target projection position; or, in the case that the first change space position does not belong to a reasonable position, adjust the first change space position to the second position belonging to a reasonable position The spatial position is changed, and based on the second positioning result and the second changed spatial position, a third projection position of the virtual object on the current shooting frame is determined as a target projection position.

In the above solution, the target projection position is determined based on the second positioning result and the original space position, or the second change space position is adjusted to a reasonable position according to the first change space position and the target projection position is determined based on the second change space position, so that the After the display position of the virtual object is corrected, the adverse effect caused by the projection error is well reduced.

In some disclosed embodiments, the first correction strategy is to directly render the virtual object to the target projection position in the next display rendering process, and the second correction strategy is to move the virtual object from the first projection position to the target projection position at a preset speed.

In the above solution, by setting different correction strategies, the correction process is more flexible.

In some disclosed embodiments, in response to obtaining the corrected second positioning result of the target device and the virtual object meets the first preset condition, before continuing to determine the first projection position as the display position of the virtual object in the current shooting frame , the determining part 41 is further configured to: use the first projection position, the projection parameters of the target device, and the first transformation parameters between the world coordinate system and the camera coordinate system of the target device to obtain the first changed space position on the world coordinate system , wherein the first transformation parameter is obtained based on the second positioning result of the target device.

In the above scheme, the first changed space position of the virtual object is determined by the second positioning result of the target device and the first projection position of the virtual object, and the projection position of the virtual object corresponding to the current shooting picture at the first changed space position remains unchanged. .

In some disclosed embodiments, the determining part 41 determines the first projection position of the virtual object on the current shooting frame of the target device based on the first positioning result of the target device before correction, including: using the original spatial position of the virtual object, the target The projection parameters of the device and the second transformation parameters between the world coordinate system and the camera coordinate system of the target device are obtained to obtain the first projection position, wherein the second transformation parameters are obtained based on the first positioning result of the target device; After the corrected second positioning result of the target device, in response to obtaining the corrected second positioning result of the target device and the virtual object meets the first preset condition, continue to determine the first projection position as the virtual object in the current shooting Before displaying the position in the screen, the fusion part 42 is further configured to: locate and match the target device with the preset map to obtain a second positioning result of the target device in the preset map.

In the above solution, the second positioning result of the target device in the preset map is obtained by positioning and matching the target device with the preset map, so that the second positioning result is more accurate than the first positioning result before correction.

Since the problem-solving principle of the device in the embodiment of the present disclosure is similar to the above-mentioned virtual fusion method in the embodiment of the present disclosure, the implementation of the device embodiment may refer to the implementation of the method embodiment.

In the embodiments of the present disclosure and other embodiments, a "part" may be a part of a circuit, a part of a processor, a part of a program or software, etc., of course it may also be a unit, a module or a non-modular one.

Please refer to FIG. 5 . FIG. 5 is a schematic structural diagram of an embodiment of the electronic device of the present disclosure. The electronic device 50 includes a memory 51 and a processor 52, and the processor 52 is configured to execute the program instructions stored in the memory 51, so as to realize the steps in the above embodiment of the virtual-real fusion method. In a specific implementation scenario, the electronic device 50 may include, but is not limited to: an AR device, a microcomputer, and a server. In addition, the electronic device 50 may also include mobile devices such as notebook computers and tablet computers, which are not limited here.

Specifically, the processor 52 is configured to control itself and the memory 51 to implement the steps in the above embodiment of the virtual-real fusion method. The processor 52 may also be called a CPU (Central Processing Unit, central processing unit). The processor 52 may be an integrated circuit chip with signal processing capability. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field to Programmable Gate Array, FPGA) or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. In addition, the processor 52 may be jointly realized by an integrated circuit chip.

In the above solution, after obtaining the corrected second positioning result of the target device, in response to the fact that the virtual object satisfies the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting screen, which can reduce the risk of the virtual object due to The visual impact caused by the jump of the display position.

Please refer to FIG. 6 . FIG. 6 is a schematic structural diagram of an embodiment of a computer-readable storage medium of the present disclosure. The computer-readable storage medium 60 stores program instructions 601. When the program instructions 601 are executed by the processor, they are configured to implement the steps in the above embodiments of the virtual-real fusion method.

In the above solution, after obtaining the corrected second positioning result of the target device, in response to the fact that the virtual object satisfies the first preset condition, the virtual object is kept displayed at the first projection position of the current shooting screen, which can reduce the risk of the virtual object due to The visual impact caused by the jump of the display position.

In some embodiments, the functions or parts included in the apparatus provided by the embodiments of the present disclosure may be configured to execute the methods described in the above method embodiments, and for specific implementation, refer to the descriptions of the above method embodiments.

This disclosure relates to the field of augmented reality. By acquiring the image information of the target object in the real environment, and then using various visual correlation algorithms to detect or identify the relevant features, states and attributes of the target object, and thus obtain the image information that matches the specific application. AR effect combining virtual and reality. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms can involve visual positioning, real-time positioning in map construction (Simultaneous Localization and Mapping, SLAM), 3D reconstruction, image registration, background segmentation, key point extraction and tracking of objects, object pose or depth detection, etc. Specific applications can not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlays and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual Interactive scenarios such as model display.

The relevant features, states and attributes of the target object can be detected or identified through the convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, and the same or similar points can be referred to each other.

Embodiments of the present disclosure may be a system, method, computer readable storage medium, and/or computer program product. The computer program product may include a computer-readable storage medium, which may be a volatile or non-volatile storage medium, on which are loaded computer-readable program instructions for causing a processor to implement various aspects of the embodiments of the present disclosure. The computer program product can be realized by hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like. A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.

In the several embodiments provided in the present disclosure, it should be understood that the disclosed methods and devices may be implemented in other ways. For example, the device implementations described above are schematic, for example, the division of parts or units is a logical function division, and there may be other division methods in actual implementation, for example, units or components can be combined or integrated into another A system, or some feature, can be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or part of the contribution to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) execute all or part of the steps of the methods in various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

Industrial Applicability

In the embodiment of the present disclosure, based on the first positioning result of the target device before correction, the first projection position of the virtual object in the current shooting frame of the target device is determined, wherein the first projection position is determined as the position of the virtual object in the current shooting frame Display position: in response to obtaining the corrected second positioning result of the target device and the virtual object meets the first preset condition, continue to determine the first projected position as the display position of the virtual object in the current shooting frame. The above solution can reduce the adverse effect caused by the jump of the virtual object after the positioning result of the target device is corrected.

Claims (29)

1. A virtual reality fusion method, comprising:

   Based on the first positioning result of the target device before correction, determine a first projection position of the virtual object on the current shooting frame of the target device, wherein the first projection position is determined as the virtual object on the current shooting frame The display position in ;

   In response to obtaining the corrected second positioning result of the target device and the virtual object satisfies a first preset condition, continue to determine the first projection position as the virtual object in the current shooting frame display position of .
2. The method according to claim 1, wherein, after determining the display position, the method further comprises:

   displaying the current shooting frame, and displaying the virtual object at the display position in the current shooting frame.
3. The method according to claim 1 or 2, wherein the first preset condition includes at least one of the following: the virtual object meets the second preset condition, the first changed space position of the virtual object is different from the original space The position deviation between the positions meets the preset position requirements, and the second preset condition includes that the virtual object belongs to the first type and/or the first change space position of the virtual object belongs to a reasonable position; wherein, the second A modified spatial position is a spatial position determined based on the second positioning result of the target device, and the first modified spatial position is projected to the first projected position.
4. The method according to claim 3, wherein the preset position requirement includes that the position deviation is smaller than a preset threshold.
5. The method according to claim 4, wherein the preset position requirement further includes that the display change process of the virtual object is visible, wherein the display change process is that the display of the virtual object changes from the first projection position to The process of projecting the location of the target.
6. The method according to any one of claims 3 to 5, wherein the first preset condition includes that the virtual item meets a second preset condition, and the second preset condition includes the first preset condition of the virtual item 1. Changing the location of the space is a reasonable location; among them,

   The reasonable position includes at least one of a non-dangerous area and a preset height range; and/or, the first projection position continues to be determined as the display of the virtual object in the current shooting frame Before the location, the method also includes:

   Based on the semantic information of the preset map, it is determined whether the first changed space position belongs to the reasonable position.
7. The method according to any one of claims 1 to 6, wherein the method further comprises:

   In response to obtaining the corrected second positioning result of the target device and the virtual object does not satisfy the first preset condition, adopting a first correction strategy or a second correction strategy to convert the The display position is changed from the first projection position to a target projection position.
8. The method according to claim 7, wherein, in response to obtaining the corrected second positioning result of the target device and the virtual object does not satisfy the first preset condition, adopting a first correction strategy Or the second correction strategy changes the display position of the virtual object from the first projection position to a target projection position, including:

   In response to the virtual object belonging to the second type, changing the display position of the virtual object from the first projected position to a target projected position by adopting the first correction strategy;

   In response to the virtual object belonging to the third type, using the second correction strategy to change the display position of the virtual object from the first projection position to a target projection position;

   In response to the first changing space position not belonging to a reasonable position, selecting the first correction strategy or the second correction strategy to change the display position of the virtual object from the first projection position to a target projection position.
9. The method according to claim 7, wherein the preset position requirement includes that the display change process of the virtual object is visible, and the position deviation is smaller than a preset threshold, wherein the display change process is the virtual object The process of the display changing from the first projection position to the target projection position;

   In response to obtaining the corrected second positioning result of the target device and the virtual object does not satisfy the first preset condition, adopting the first correction strategy or the second correction strategy to change the position of the virtual object to The display position is changed from the first projection position to the target projection position, including:

   In response to the fact that the virtual object meets the second preset condition, the display change process of the virtual object is visible, and the position deviation is not less than the preset threshold, using the first correction strategy to restore the virtual object to The display position of is changed from the first projection position to the target projection position;

   In response to the fact that the virtual object meets the second preset condition and the display change process of the virtual object is invisible, using the second correction strategy to change the display position of the virtual object from the first projection position at the target projection position.
10. The method according to any one of claims 7 to 9, wherein, when the first correction strategy or the second correction strategy is used to change the display position of the virtual object from the first projection position to the target projection Before the location, the method also includes:

    Based on the second positioning result and the original spatial position, determine a second projection position of the virtual object on the current shooting frame as the target projection position; or,

    If the first changed space position does not belong to a reasonable position, adjust the first changed space position to a second changed space position belonging to the reasonable position, based on the second positioning result and the second The spatial position is changed, and a third projection position of the virtual object on the current shooting frame is determined as the target projection position.
11. The method according to any one of claims 7 to 9, wherein the first correction strategy is to directly render the virtual object to the target projection position in the next display rendering process, and the second correction strategy is The virtual object is moved from the first projection position to the target projection position at a preset speed.
12. The method according to any one of claims 1 to 11, wherein, in response to obtaining the corrected second positioning result of the target device and the virtual object satisfies the first preset condition, the The first projection position continues to be determined as before the display position of the virtual object in the current shooting frame, and the method further includes:

    Using the first projection position, projection parameters of the target device, and first transformation parameters between the world coordinate system and the camera coordinate system of the target device, to obtain the first modification on the world coordinate system The spatial position, wherein the first transformation parameter is obtained based on the second positioning result of the target device.
13. The method according to any one of claims 1 to 12, wherein, based on the first positioning result of the target device before correction, determining the first projection position of the virtual object on the current shooting picture of the target device includes:

    Using the original spatial position of the virtual object, the projection parameters of the target device, and the second transformation parameters between the world coordinate system and the camera coordinate system of the target device, the first projection position is obtained, wherein the The second transformation parameter is obtained based on the first positioning result of the target device;

    In response to obtaining the corrected second positioning result of the target device and the virtual object satisfies a first preset condition, continue to determine the first projection position as the virtual object in the current shooting Before displaying the position in the picture, the method also includes:

    Positioning and matching the target device with a preset map to obtain the second positioning result of the target device in the preset map.
14. A virtual reality fusion device, comprising:

    The determining part is configured to determine a first projection position of the virtual object on the current shot frame of the target device based on the first positioning result of the target device before correction, wherein the first projection position is determined as the virtual object at The display position in the current shooting picture;

    The fusion part is configured to, in response to obtaining the corrected second positioning result of the target device and the virtual object meets a first preset condition, continue to determine the first projection position as the virtual object at Describe the display position in the current shooting screen.
15. The device according to claim 14, wherein, after determining the display position, the fusion part is further configured to: display the current shooting frame, and display the dummy.
16. The device according to claim 14 or 15, wherein the first preset condition includes at least one of the following: the virtual object meets the second preset condition, the first changed space position of the virtual object is different from the original space The position deviation between the positions meets the preset position requirements, and the second preset condition includes that the virtual object belongs to the first type and/or the first change space position of the virtual object belongs to a reasonable position; wherein, the second A modified spatial position is a spatial position determined based on the second positioning result of the target device, and the first modified spatial position is projected to the first projected position.
17. The apparatus according to claim 16, wherein the preset position requirement includes that the position deviation is smaller than a preset threshold.
18. The device according to claim 17, wherein the preset position requirement further includes that the display change process of the virtual object is visible, wherein the display change process is that the display of the virtual object changes from the first projection position to The process of projecting the location of the target.
19. The device according to any one of claims 16 to 18, wherein the first preset condition includes that the virtual item meets a second preset condition, and the second preset condition includes the first preset condition of the virtual item. A changed spatial position belongs to a reasonable position; wherein, the reasonable position includes at least one of a non-dangerous area and a preset height range; and/or, the first projected position continues to be determined as the virtual Before the display position of the object in the current shooting frame, the fusion part is further configured to: determine whether the first changed spatial position belongs to the reasonable position based on semantic information of a preset map.
20. The apparatus according to any one of claims 14 to 19, wherein the fusion part is further configured: in response to obtaining the corrected second positioning result of the target device, and the virtual object does not satisfy the The first preset condition is to change the display position of the virtual object from the first projection position to a target projection position by adopting a first correction strategy or a second correction strategy.
21. The apparatus according to claim 20, wherein the fusion part adopts a first changing the display position of the virtual object from the first projection position to a target projection position by a correction strategy or a second correction strategy, comprising: adopting the first correction strategy in response to the virtual object belonging to the second type changing the display position of the virtual object from the first projection position to a target projection position; in response to the virtual object belonging to a third type, adopting the second correction strategy to change the display position of the virtual object from the The first projection position is changed to the target projection position; in response to the first changed space position not belonging to a reasonable position, selecting the first correction strategy or the second correction strategy to change the display position of the virtual object from the first A projection position is changed to the target projection position.
22. The device according to claim 20, wherein the preset position requirement includes that the display change process of the virtual object is visible, and the position deviation is smaller than a preset threshold, wherein the display change process is the virtual object The process of displaying the display from the first projection position to the target projection position; the fusion part responds to obtaining the corrected second positioning result of the target device, and the virtual object does not satisfy the first preset condition Using a first correction strategy or a second correction strategy to change the display position of the virtual object from the first projection position to a target projection position, including:

    In response to the fact that the virtual object meets the second preset condition, the display change process of the virtual object is visible, and the position deviation is not less than the preset threshold, using the first correction strategy to restore the virtual object to The display position of the virtual object is changed from the first projection position to the target projection position; in response to the fact that the virtual object meets the second preset condition and the display change process of the virtual object is invisible, the second correction strategy is adopted Changing the display position of the virtual object from the first projection position to a target projection position
23. The device according to any one of claims 20 to 22, wherein when the first correction strategy or the second correction strategy is used to change the display position of the virtual object from the first projection position to the target projection Before the position, the fusion part further includes determining a second projection position of the virtual object in the current shot frame as the target projection position based on the second positioning result and the original spatial position; or,

    If the first changed space position does not belong to a reasonable position, adjust the first changed space position to a second changed space position belonging to the reasonable position, based on the second positioning result and the second The spatial position is changed, and a third projection position of the virtual object on the current shooting frame is determined as the target projection position.
24. The device according to any one of claims 20 to 22, wherein the first correction strategy is to directly render the virtual object to the target projection position in the next display rendering process, and the second correction strategy is The virtual object is moved from the first projection position to a target projection position at a preset speed.
25. The device according to any one of claims 14 to 24, wherein, in response to obtaining the corrected second positioning result of the target device and the virtual object satisfies the first preset condition, the Before the first projection position continues to be determined as the display position of the virtual object in the current shooting frame, the determination part is further configured to use the first projection position, the projection parameters of the target device, and the world coordinates system and the camera coordinate system of the target device to obtain the first changed space position on the world coordinate system, wherein the first transformation parameter is based on the second The positioning result is obtained.
26. The device according to any one of claims 14 to 25, wherein the determining part determines the first projection position of the virtual object on the current shooting frame of the target device based on the first positioning result of the target device before correction, The method includes: obtaining the first projected position by using the original spatial position of the virtual object, the projection parameter of the target device, and the second transformation parameter between the world coordinate system and the camera coordinate system of the target device, wherein , the second transformation parameter is obtained based on the first positioning result of the target device;

    In response to obtaining the corrected second positioning result of the target device and the virtual object meets the first preset condition, continue to determine the first projection position as the virtual object in the current shooting frame Before displaying the position in , the fusion part is further configured to perform positioning and matching of the target device with a preset map to obtain the second positioning result of the target device in the preset map.
27. An electronic device, comprising a memory and a processor, the processor configured to execute program instructions stored in the memory, so as to implement the method according to any one of claims 1 to 13.
28. A computer-readable storage medium, on which program instructions are stored, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 13 is realized.
29. A computer program product, comprising computer readable code, when the computer readable code is run in an electronic device, the processor in the electronic device executes and is configured to implement the method described in any one of claims 1 to 13 method.

Images