WO2022205026A1

WO2022205026A1 - Product display method and apparatus, and electronic device

Info

Publication number: WO2022205026A1
Application number: PCT/CN2021/084247
Authority: WO
Inventors: 程龙一
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2021-03-31
Filing date: 2021-03-31
Publication date: 2022-10-06

Abstract

Embodiments of the present application provide a product display method, comprising: obtaining in real time a scene image of a real scene by means of a camera; displaying, at a position in a pre-constructed virtual scene corresponding to a designated position in the real scene, a virtual element used for assembling a first virtual model corresponding to the product, and displaying an unfolding structural special effect of the first virtual model at a first preset relative position of the virtual element; and rendering the virtual scene in the scene image, adjusting display content of the virtual scene on the basis of obtained operation information inputted by a user, and re-rendering the adjusted virtual scene in the scene image. The embodiments of the present application further provide a product display apparatus and an electronic device.

Description

Product display method, device and electronic device

technical field

The present application relates to the field of augmented reality technology, and in particular, to a product display method, a product display device, and an electronic device.

Background technique

At present, users learn about products mainly by inquiring about the relevant information of the products on the Internet, or by visiting the physical store. However, learning about the products from the Internet cannot allow users to experience the products at close range. It is difficult to understand the internal structure of the product on display, and it is difficult to experience the assembly process of the product.

SUMMARY OF THE INVENTION

In a first aspect, the embodiments of the present application provide a product display method, including:

Real-time acquisition of scene images of real scenes through cameras;

In the pre-built virtual scene, a virtual element for assembling a first virtual model corresponding to the product is displayed at a position corresponding to a specified position in the real scene, and the virtual element is displayed at a first preset relative position of the virtual element. Describe the unfolding structure special effects of the first virtual model;

The virtual scene is rendered in the scene image, the display content of the virtual scene is adjusted based on the obtained operation information input by the user, and the adjusted virtual scene is re-rendered in the scene image.

In a second aspect, an embodiment of the present application provides a product display device, including:

processor;

memory for storing processor executable program instructions;

The processor implements the following operations when executing the executable program instructions:

Real-time acquisition of scene images of real scenes through cameras;

In a third aspect, an embodiment of the present application provides an electronic device, including:

camera, for capturing images;

processor;

memory for storing processor executable program instructions;

Real-time acquisition of scene images of real scenes through cameras;

In the product display method provided by the embodiment of the present application, as an improvement, the virtual elements used to form the first virtual model corresponding to the product and the first virtual elements corresponding to the product are rendered and displayed on the scene image of the real scene obtained by the camera in real time. A special effect of the unfolding structure of the virtual model, so that the user can see the unfolding structure of the product displayed in the real scene and the components of the product through the scene image in real time, so that the user can experience the real display of the product no matter where the user is. The visual effects in front of their own eyes also allow users to get close contact to understand the specific shape of the product. Further, the user can also freely operate the components of the displayed product, such as assembling, thereby allowing the user to experience the assembly process of the product, and because the unfolding structure of the product is also displayed at the same time, it can help the user more A clear and visual understanding of the structure and composition of the product, and the ability to assemble components with reference to the unfolded structure of the displayed product.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the drawings that are used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

FIG. 1 is a flow chart of a product display method according to an exemplary embodiment of the present application.

FIG. 2 is a schematic diagram of a scene of a product display according to an exemplary embodiment of the present application.

Figure 3a is a scene image of a real scene captured by a camera in real time.

Figure 3b is a picture output and displayed by the electronic device after the camera completes the plane information detection of the real scene.

Fig. 3c is a schematic diagram of a user selecting a landing point in a scene image by clicking on the screen with a finger.

FIG. 3d is a screen output by the electronic device after the user clicks and selects a landing point in the scene image.

FIG. 3e is a screen output and displayed by the electronic device after the user clicks on the virtual package.

FIG. 3f is a screen output and displayed by the electronic device after the user clicks and selects a certain component.

FIG. 3g is a schematic diagram of the user selecting the component and sliding the screen to rotate the component.

Figure 3h is a schematic diagram of the user completing a component assembly.

Figure 3i is a schematic diagram of the user selecting another component on the screen.

FIG. 3j is a schematic diagram of the user clicking to expand the area of the corresponding component in the structure special effect.

Figure 3k is a schematic diagram of the user completing another component assembly.

FIG. 31 is a screen output and displayed by the electronic device after the user assembles.

FIG. 3m is a schematic diagram of a control component composed of virtual joysticks and virtual buttons rendered on the screen of an electronic device.

FIG. 4 is a schematic diagram of a product display device according to an exemplary embodiment of the present application.

FIG. 5 is a schematic diagram of an electronic device according to an exemplary embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application. It can be understood that the various embodiments presented in this application can be combined with each other without conflict.

In some cases, for some products that may need to be assembled by the user, they are often displayed in the complete assembled form when displayed in the physical store, and it is difficult for the user to clearly understand the specific structure and composition of the product even in the physical store. In addition, users may often rely on paper manuals, teaching videos or manual explanations with text and pictures to assemble products. It is difficult for users to visually understand and memorize the components that make up the product and the assembly relationship between the components. Therefore, it is currently required. A way to help users better understand and assemble products.

In view of at least the above problems, an embodiment of the present application first proposes a product display method. Referring to FIG. 1 , FIG. 1 is a flowchart of a product display method shown in an exemplary embodiment of the present application, and the method includes the following steps:

S101, obtain a scene image of a real scene in real time through a camera;

S102, displaying a virtual element used to assemble a first virtual model corresponding to the product at a position corresponding to a specified position in the real scene in the pre-built virtual scene, and at a first preset relative position of the virtual element displaying the special effects of the expanded structure of the first virtual model;

S103: Render the virtual scene in the scene image, adjust the display content of the virtual scene based on the obtained operation information input by the user, and re-render the adjusted virtual scene in the scene image.

The methods of the above embodiments of the present application can be applied to electronic devices, such as mobile phones, tablets, notebook computers, AR (Augmented Reality, augmented reality) glasses and other electronic devices. The camera can be equipped with a camera for capturing images, or real-time image capturing can be performed through an external camera, which is not limited. The electronic device may also be equipped with a display screen for outputting and displaying the rendered scene image, so that the user can view the rendered scene image through the screen.

The product display method provided by the above embodiments of the present application, as an improvement, renders and displays the virtual elements used to form the first virtual model corresponding to the product on the scene image of the real scene acquired by the camera in real time, and the product corresponds to The special effects of the unfolded structure of the first virtual model of the virtual model allow the user to see the unfolded structure of the product displayed in the real scene and the components of the product through the scene image in real time, so that the user can feel as if the user is wherever he is. The product is displayed in front of their own eyes, so that users can get a close contact to understand the specific shape of the product. Further, the user can also freely operate the components of the displayed product, such as assembling them into a product, thereby allowing the user to experience the assembly process of the product, and because the unfolded structure of the product is also displayed at the same time. , which can help the user to understand the structure and composition of the product more clearly and vividly, and the user can also assemble the components with reference to the unfolded structure of the displayed product.

The product display method provided by the above embodiments of the present application will be further introduced below. Among them, rendering the virtual scene in the scene image can be realized by technologies such as ARkit, ARcore, etc., such technologies can realize the placement of 2D or 3D elements (text, pictures, models, audio and video, etc.) in the image collected by the device camera, Thereby creating an illusion that the virtual elements actually exist in the real world. Specifically, taking ARKit as an example, it integrates technologies such as device motion tracking, camera image acquisition, image visual processing, and scene rendering, and provides an easy-to-use API (Application Programming Interface) to facilitate developers to develop AR application, developers can no longer pay attention to the underlying technical implementation details, which reduces the difficulty of AR application development to a certain extent. It can be understood that the embodiments of the present application may use the above-mentioned technologies as the basis for realizing product display, and of course it is not limited to this. If feasible, the embodiments of the present application may also adopt other means. This is not listed one by one.

Exemplarily, the virtual scene may be a virtual space, and a specific method of rendering the virtual scene in the scene image may be to first establish a reference coordinate system at a position corresponding to a specified position in the real scene in the pre-built virtual scene, The virtual scene is then rendered in the scene image based on the transformation relationship between the reference coordinate system and the image coordinate system of the scene image. Through the establishment of the reference coordinate system, the origin of the reference coordinate system in the virtual scene can be used as the reference point corresponding to the specified position in the real scene, so that any position in the virtual scene can correspond to the position in the real scene. When the virtual scene is rendered into the scene image based on the conversion relationship between the reference coordinate system and the image coordinate system of the scene image, it is equivalent to superimposing the two-dimensional image of the virtual scene on the scene image of the real scene. The position corresponds to the position in the real scene, so that the final rendered scene image can show the visual effects that the virtual model, special effects, etc. placed in the virtual scene seem to exist in the real world.

The products mentioned in the embodiments of the present application may be arbitrarily manufactured products, especially some products that may require certain assembly after purchase by the user, such as drones, unmanned vehicles, robots, and other types of products. As mentioned in S102, the first virtual model corresponding to the product may be a virtual model constructed with reference to the external structure, components, etc. of the product, and the first virtual model may be completely consistent with the product. Of course, due to the product structure The complexity of the model and the difficulty of modeling can allow a certain structural error between the first virtual model and the product, which the technician can choose according to the actual needs during modeling. The special effect of the expanded structure of the first virtual model may specifically be used to represent the virtual elements (ie, components) included in the first virtual model (ie, the product) and the relative positional relationship between the virtual elements (ie, the assembly between the components). relationship) effects. In addition, for the first preset relative position mentioned in S102, it can be any relative position of the virtual element, for example, it can be to render and display the special effect of the expanded structure above the virtual element, of course, it is not limited to this. Since any position in the pre-built virtual scene corresponds to the position in the real scene, by displaying the virtual element used to assemble the first virtual model corresponding to the product at the position in the virtual scene corresponding to the specified position in the real scene, and displaying the special effects of the unfolding structure of the first virtual model at the first preset relative position of the virtual element, and then rendering the virtual scene on the scene image, the first virtual corresponding to a product can be displayed on the scene image. Model and unfolding structure effects are visual effects that actually exist in the real world where the user lives. Further users can also freely manipulate virtual elements in the virtual scene rendered on the scene image, such as assembling virtual elements. The device applying the method can adjust the position, posture, etc. of the virtual elements displayed in the virtual scene according to the operation information input by the user, and re-render the adjusted virtual scene on the scene image obtained in real time, so as to present the user in the real world. The effect of assembling virtual elements allows users to experience a more realistic product assembly process.

In order to more vividly illustrate the product display effect of the product display method provided by the embodiment of the present application, reference may be made to FIG. 2 . FIG. 2 is a schematic diagram of a product display scene shown in an exemplary embodiment of the present application. A virtual element 201 , and an expanded structure 202 displayed above the virtual element 201 .

In order to make the visual effect of displaying the product in the real scene more realistic, in one embodiment, after obtaining the scene image of the real scene through the camera in real time, the plane information in the real scene can also be detected, and the plane information in the virtual scene can be determined based on the plane information. plane. By detecting the plane information in the real scene, the plane in the real scene can be determined, and then the plane in the virtual scene can be determined accordingly, so that if the specified position in the real scene is on a certain plane, the plane in the virtual scene can be determined by The virtual elements are displayed on the corresponding plane positions, and then the virtual scene is rendered on the scene image, which can more realistically display the visual effect of the virtual elements placed on a specified position on a certain plane in the real scene on the scene image.

For the specified position in the real scene, it may be the position in the real scene corresponding to the position selected by the user in the scene image obtained by the camera in real time; or it may be the position in the real scene corresponding to the random position in the scene image ; Or it can also be a location with a logo in the real scene, etc. How to set the designated position in the real scene specifically, the technician can select it according to the actual needs, which is not limited. The description is given by taking as an example that the specified position in the real scene is the position in the real scene corresponding to the position selected by the user in the scene image obtained by the camera in real time.

In one embodiment, if the specified position is the position in the real scene corresponding to the position selected by the user in the scene image obtained by the camera in real time, the display device is displayed at the position corresponding to the specified position in the real scene in the pre-built virtual scene. Before assembling into the virtual element of the first virtual model corresponding to the product, the position selected by the user in the scene image can also be obtained, and the position in the real scene corresponding to the position selected by the user in the scene image is determined as the Specify the location. Further, in order to more accurately identify the plane position corresponding to the position selected by the user in the scene image in the real scene, the plane information in the real scene can also be detected with reference to the previous embodiment, so as to identify the collected scene. Plane information on the image.

In order to prompt the user to select a position in the scene image, a prompt special effect for prompting the user to select a position in the scene image can also be rendered in the scene image. The form of the prompt effect can be any text, picture, virtual model, etc. One form or a combination of multiple forms can be set by technical personnel according to requirements, which is not limited.

In addition, after obtaining the position selected by the user in the scene image according to the prompt effect,

It is possible to stop rendering the prompting special effect in the scene image, and perform S102 and S103 described in the previous embodiment; or it can also stop rendering the prompting special effect in the scene image, and correspond to the real scene in the pre-built virtual scene. The second virtual model is displayed at the position of the specified position, and the virtual scene is rendered in the scene image, that is, the second virtual model is displayed at the position selected by the user in the scene image, so that the second virtual model is displayed on the scene image. A visual effect placed at a user-specified location in the user's real world. The second virtual model may be any type of virtual model, for example, a virtual box, a virtual box, and the like. Preferably, the second virtual model may be a virtual box used to represent a product to be displayed inside, and while rendering the virtual box, a prompting text may be added to the scene image to prompt the user to obtain the product successfully. Product, enter the operation information to display the product, for example, a specific text form can be "Wow! Get XX product! Please click to open the box". On this basis, further, displaying the virtual elements used to assemble the first virtual model corresponding to the product at a position corresponding to a specified position in the real scene in the pre-built virtual scene may During the selection operation of the second virtual model, the second virtual model is rendered at a position corresponding to the specified position in the real scene in the pre-built virtual scene as a virtual image for assembling the first virtual model corresponding to the product element. In this way, a visual effect of unboxing a product can be shown to the user, making the user experience more realistic.

The following describes the product display method provided by the embodiment of the present application in combination with the operation information input by the user. In one embodiment, in S103, adjusting the display content of the virtual scene based on the obtained operation information input by the user may be, moving the position of the virtual element in the virtual scene in real time based on the operation information input by the user, of course, at the same time, The pose of the virtual element is changed based on the operation information input by the user. This allows users to freely manipulate virtual elements to assemble them. Wherein, in order to make the user's assembling process more convenient, the user can assemble the virtual elements based on the special effects of the unfolded structure of the first virtual model. Each component of the model, so in fact, each virtual element has a corresponding area on the expanded structure effect. When the user moves the virtual element to the area corresponding to the virtual element in the expanded structure effect, the virtual element can be fixed and displayed in this area. area. Therefore, if all the virtual elements are moved by the user to the area corresponding to the virtual elements in the unfolded structure special effect, the unfolded structure special effect can be rendered into the first virtual model, thereby completing the assembly process of the product. Such an assembly process is relatively simple for the user, which not only helps the user experience the simple assembly process, but also enables the user to more clearly understand the structural composition of the product.

Among them, since the camera represents the viewing angle of the user observing the virtual elements and special effects in the virtual scene, the conversion relationship between the reference coordinate system in the virtual scene and the image coordinate system of the scene image varies with the relative pose of the camera and the virtual scene. Therefore, the relative pose between the camera and the virtual scene can also be acquired in real time, so that when the shooting angle of the camera changes, the virtual scene can be rendered on the scene image collected in real time according to the changed perspective.

Further, moving the position of the virtual element in the virtual scene in real time based on the operation information input by the user may include determining the selected virtual element based on the operation information input by the user, and rendering the selected virtual element in the virtual scene of the camera. at the second preset relative position. The operation information input by the user may be the user's operation of clicking and selecting the virtual element rendered on the scene image, and the second preset relative position may be any relative position of the camera. Preferably, it may be the one in front of the camera that is close to the camera. The position, more preferably, can be the position of the preset distance directly in front of the camera, similar to the effect of "sucking" in front of the lens. Therefore, when the user clicks to select a certain virtual element, the virtual element can be fixed and rendered at the center of the scene image as the visual focus, and its position on the scene image will not change with the change of the camera view. On this basis, when the position of the camera in the virtual scene changes, the position of the selected virtual element in the virtual scene can be changed accordingly, so that the selected virtual element can be moved by changing the pose of the camera in the virtual scene. The position of the virtual element in the virtual scene. In addition, the position of the selected virtual element in the virtual scene may be moved based on the position selected by the user in the scene image. For example, for the virtual element selected by the user in the scene image, the position of the selected virtual element in the scene image may be moved The selected position determines the corresponding position in the virtual scene and moves the virtual element to that position.

In one embodiment, when the user selects a virtual element, the brief information of the selected virtual element may be added to the scene image for output display. The brief information displayed in the output may include the basic information of the components that make up the product corresponding to the virtual element, such as one or more kinds of information such as name, material, color, use, size, etc. Get a comprehensive understanding of the part's information and its role in the product.

In one embodiment, when the user selects a virtual element, the posture of the selected virtual element at the second preset relative position of the camera may be further changed based on other operation information input by the user, wherein the user Other input operation information may include the user's operation of selecting a virtual element in the displayed scene image on the display screen of the device, and a sliding operation in any direction on the screen while selecting the virtual element. The posture of the selected virtual element in the virtual scene is changed by the sliding operation, so that the user can rotate the virtual element by selecting and sliding on the display screen, so as to observe the structure of the virtual element from a full perspective.

In one embodiment, on the basis that the user assembles the virtual element by moving the virtual element to the area corresponding to the virtual element in the expanded structure special effect, when the user selects the virtual element, in the expanded structure special effect Rendering the region identifier corresponding to the selected virtual element, for example, it can be to highlight, flash, expand, etc., the region corresponding to the selected virtual element in the expanded structure effect, so as to highlight the virtual element corresponding to the user selected virtual element in the expanded structure effect. area, which is convenient for users to quickly find the target area for assembly.

In one embodiment, after the user completes assembling the virtual elements to obtain the first virtual model, a control component may be further rendered on the display screen of the electronic device for the user to control the movement of the first virtual model. Since what the user sees from the electronic device is that the first virtual model is moving in the real scene, but in fact the first virtual model is moving in the virtual scene, in order to make the user's operating experience more realistic, the previous implementation can be combined with For example, the plane information in the real scene is detected to determine the plane in the virtual scene, so that the user can control the first virtual model to move based on the plane information in the real scene.

So far, the introduction of the product display method provided by the embodiments of the present application has been completed. In order to visualize the methods introduced by the above embodiments of the present application, an actual example is given here for further explanation in conjunction with the diagrams and the description text. It can be understood that the following The given examples are for illustrative purposes only, and are not intended to be limiting.

Referring to Fig. 3a, Fig. 3a is a scene image of a real scene collected in real time by a camera on the electronic device called by the application when the user opens a specified application on the electronic device, and the camera detects the plane information of the real scene while collecting the image, for example, a real scene. The ground in the scene, and the electronic device outputs and displays the acquired scene image through the display screen, and below the output and displayed scene image is also added explanatory text for introducing the application.

Referring to Fig. 3b, Fig. 3b is the screen output and displayed by the electronic device after the camera completes the plane information detection of the real scene, wherein the transparent carton special effect rendered on the scene image of the real scene is displayed in the middle of the screen, and the bottom of the scene image is output and displayed with The prompt text to guide the user to select the drop point.

Referring to Fig. 3c, Fig. 3c is a schematic diagram showing that the user selects a landing point in the scene image by clicking the screen with a finger.

Referring to Fig. 3d, Fig. 3d is a screen output by the electronic device after the user clicks and selects the drop point in the scene image. Specifically, the electronic device renders the animation effect of the virtual packaging box falling from the sky, and at the same time, the virtual packaging is rendered at the drop point selected by the user. box, and output and display the prompt text for prompting the user to obtain the XX product, and prompting the user to click on the virtual packaging box to display the product.

Referring to Fig. 3e, Fig. 3e is a picture displayed by the electronic device after the user clicks on the virtual packaging box. The upper part of the picture shows the special effects of the unfolding structure of the product rendered on the scene image, and the lower part shows the corresponding product for assembling into a product. The parts of the model, together with the text description prompting the user to assemble the parts, as well as the progress and time of the user's assembly.

Referring to Fig. 3f, Fig. 3f is the screen output and displayed by the electronic device after the user clicks and selects a certain part, in which the part selected by the user is rendered in the center of the scene image, and the explanatory text introducing the basic information of the part is displayed at the bottom of the screen, and at the same time. The area of the expanded structure effect corresponding to the user-selected component is highlighted. Further, the user can slide the screen to rotate the component while selecting the component, the schematic diagram of which can be seen in Figure 3g. At the same time, the user can move the selected component to the corresponding area in the unfolded structure effect by holding the electronic device in the real scene to complete the assembly. Refer to FIG. 3h for a schematic diagram showing the assembly of a component.

Referring to Fig. 3i, Fig. 3i is a schematic diagram of another component selected by the user on the screen. Similarly, the component selected by the user is rendered in the center of the scene image, and an explanatory text introducing the basic information of the component is displayed at the bottom of the screen. The area corresponding to the component selected by the user in the expanded structure effect is highlighted. At this time, the user can also click to expand the area corresponding to the component in the structural effect to complete the assembly of the component. Refer to Figure 3j and Figure 3k for the schematic diagrams of the area corresponding to the component in the user click to expand the structural effect, and Schematic diagram of the completed assembly of this part.

Referring to Fig. 31, Fig. 31 is the picture of the output display of the electronic device after the assembly is completed, wherein the unfolded structural special effects and each component are rendered as the model corresponding to the product, further referring to Fig. 3m, the screen is rendered by a virtual joystick and a virtual button. Consists of control components for the user to control the movement of the manipulator model.

An embodiment of the present application further provides a product display device. Referring to FIG. 4, FIG. 4 is a schematic diagram of a product display device shown in an exemplary embodiment of the present application, including:

processor 401;

a memory 402 for storing program instructions executable by the processor 401;

When the processor 401 executes the executable program instructions, the following operations are implemented:

Real-time acquisition of scene images of real scenes through cameras;

It can be understood that, on the basis of the processor and memory shown in the above-mentioned product display device, the product display device may also include other components necessary for normal operation according to its actual type. For example, in the case where the product display device is a processing chip or a motherboard integrated with a processing chip, it can realize the control function by being installed on the device, for example, installed on the electronic device described in the previous embodiment. At this time, the product display device can also be Including a communication interface for data interaction with other devices on the device; in the case where the product display device is an electronic device, the product display device may also include, for example, an input/output interface, a communication interface, a bus, etc., input/output Interfaces can be used to connect input/output modules to realize information input and output. The input/output/module can be configured in the electronic device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various types of sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, and the like. The communication interface is used to connect a communication module (not shown in the figure) to realize communication interaction between the electronic device and other devices. The communication module may implement communication through wired means (eg, USB, network cable, etc.), or may implement communication through wireless means (eg, mobile network, WIFI, Bluetooth, etc.). A bus includes a path that transfers information between various components of an electronic device, such as the processor, memory, input/output interfaces, and communication interfaces.

An embodiment of the present application further provides an electronic device. Referring to FIG. 5, FIG. 5 is a schematic diagram of an electronic device shown in an exemplary embodiment of the present application, including:

a camera 501, used for capturing images;

processor 502;

a memory 503 for storing program instructions executable by the processor 502;

input/output interface 504;

communication interface 505;

bus 506;

When the processor 502 executes the executable program instructions, the following operations are implemented:

Real-time acquisition of scene images of real scenes through cameras;

The embodiments of the present application further provide a computer storage medium, on which a computer program is stored, and when the computer program is executed, the method described in any of the foregoing embodiments is implemented. For the apparatus embodiments, since they basically correspond to the method embodiments, reference may be made to the partial descriptions of the method embodiments for related parts. The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

All the embodiments given in this application may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the procedures or functions according to the embodiments of the present application are generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device. Computer instructions may be stored in or transmitted over a computer-readable storage medium. Computer instructions may be sent from one website site, computer, server, or data center to another website site, computer, via wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.) , server or data center for transmission. A computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, or the like that contains one or more of the available mediums integrated. Useful media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state disks (SSDs)), and the like.

It should be noted that, in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any relationship between these entities or operations. any such actual relationship or sequence exists. The terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also other not expressly listed elements, or also include elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The methods and devices provided by the embodiments of the present application have been introduced in detail above, and specific examples are used to illustrate the principles and implementations of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as a limitation to the application. .

Claims

A product display method, comprising:

Real-time acquisition of scene images of real scenes through cameras;

In the pre-built virtual scene, a virtual element for assembling a first virtual model corresponding to the product is displayed at a position corresponding to a specified position in the real scene, and the virtual element is displayed at a first preset relative position of the virtual element. Describe the unfolding structure special effects of the first virtual model;

The virtual scene is rendered in the scene image, the display content of the virtual scene is adjusted based on the obtained operation information input by the user, and the adjusted virtual scene is re-rendered in the scene image.
The method according to claim 1, wherein the acquiring the scene image of the real scene through the camera in real time comprises:

Detect plane information in real scenes;

A plane in the virtual scene is determined based on the plane information.
The method according to claim 1, wherein the virtual elements for assembling into the first virtual model corresponding to the product are displayed at a position corresponding to a specified position in the real scene in the pre-built virtual scene, before include:

The position selected by the user in the scene image is acquired, and the position in the real scene corresponding to the position selected by the user in the scene image is determined as the designated position.
The method according to claim 3, wherein the obtaining of the position selected by the user in the scene image includes:

Renders a prompt effect in the scene image that prompts the user to select a location in the scene image.
The method according to claim 4, wherein the acquiring the position selected by the user in the scene image further comprises:

Stop rendering the prompt special effect in the scene image, and display a second virtual model in the pre-built virtual scene at a position corresponding to the specified position in the real scene, and render the virtual scene in the scene image;

The virtual elements for assembling into the first virtual model corresponding to the product are displayed at the position corresponding to the specified position in the real scene in the pre-built virtual scene, including:

When the user's selection operation on the second virtual model is received, the second virtual model is rendered in the pre-built virtual scene at a position corresponding to the specified position in the real scene, for assembling into the corresponding product of the product. A virtual element of the first virtual model.
The method according to claim 1, wherein the rendering the virtual scene in the scene image comprises:

A reference coordinate system is established at the position corresponding to the specified position in the real scene in the pre-built virtual scene;

The virtual scene is rendered in the scene image based on the transformation relationship between the reference coordinate system and the image coordinate system of the scene image.
The method according to claim 1, wherein the adjusting the display content of the virtual scene based on the obtained operation information input by the user comprises:

The position of the virtual element in the virtual scene is moved in real time based on the operation information input by the user. If the virtual element moves to the area corresponding to the virtual element in the special effect of the unfolded structure, the virtual element is fixedly displayed in the virtual scene. area.
The method according to claim 7, wherein the method further comprises obtaining the relative pose of the camera and the virtual scene;

The moving the position of the virtual element in the virtual scene in real time based on the operation information input by the user includes:

The selected virtual element is determined based on the operation information, and the selected virtual element is rendered at a second preset relative position of the camera in the virtual scene.
The method according to claim 8, wherein the moving the position of the virtual element in the virtual scene in real time based on the operation information input by the user further comprises:

The position of the selected virtual element in the virtual scene is moved based on the pose change of the camera in the virtual scene.
The method according to claim 8, wherein the moving the position of the virtual element in the virtual scene in real time based on the operation information input by the user comprises:

Moves the position of the selected virtual element in the virtual scene based on the position selected by the user in the scene image.
The method according to claim 8, wherein the method further comprises:

The output displays brief information about the selected virtual element.
The method according to claim 8, wherein the method further comprises:

The posture of the selected virtual element at the second preset relative position of the camera is changed based on the operation information.
The method according to claim 8, wherein the method further comprises:

The region identifier corresponding to the selected virtual element is rendered in the expanded structure special effect.
The method according to claim 7, wherein the method further comprises:

If all the virtual elements are moved to the area corresponding to the virtual elements in the expanded structure effect, the expanded structure effect is rendered as the first virtual model.
The method according to claim 14, wherein the method is applied to an electronic device, and the method further comprises:

A control component is rendered on the display screen of the electronic device for the user to control the movement of the first virtual model.
The method according to claim 15, wherein the first virtual model moves based on a plane of a virtual scene; and the plane of the virtual scene is determined based on plane information of a real scene.
A product display device, characterized in that it includes:

processor;

memory for storing processor executable program instructions;

The processor implements the following operations when executing the executable program instructions:

Real-time acquisition of scene images of real scenes through cameras;

In the pre-built virtual scene, a virtual element for assembling a first virtual model corresponding to the product is displayed at a position corresponding to a specified position in the real scene, and the virtual element is displayed at a first preset relative position of the virtual element. Describe the unfolding structure special effects of the first virtual model;

The virtual scene is rendered in the scene image, the display content of the virtual scene is adjusted based on the obtained operation information input by the user, and the adjusted virtual scene is re-rendered in the scene image.
The apparatus of claim 17, wherein the processor implements the following operations when executing the executable program instructions:

Detect plane information in real scenes;

A plane in the virtual scene is determined based on the plane information.
The apparatus of claim 18, wherein the processor implements the following operations when executing the executable program instructions:

The position selected by the user in the scene image is acquired, and the position in the real scene corresponding to the position selected by the user in the scene image is determined as the designated position.
The apparatus of claim 19, wherein the processor implements the following operations when executing the executable program instructions:

Renders a prompt effect in the scene image that prompts the user to select a location in the scene image.
The apparatus according to claim 20, wherein the processor implements the following operations when executing the executable program instructions:

Stop rendering the prompt special effect in the scene image, and display a second virtual model in the pre-built virtual scene at a position corresponding to the specified position in the real scene, and render the virtual scene in the scene image;

The processor also implements the following operations when executing the executable program instructions:

When the user's selection operation on the second virtual model is received, the second virtual model is rendered in the pre-built virtual scene at a position corresponding to the specified position in the real scene, for assembling into the corresponding product of the product. A virtual element of the first virtual model.
The apparatus of claim 17, wherein the processor implements the following operations when executing the executable program instructions:

A reference coordinate system is established at the position corresponding to the specified position in the real scene in the pre-built virtual scene;

The virtual scene is rendered in the scene image based on the transformation relationship between the reference coordinate system and the image coordinate system of the scene image.
The apparatus of claim 17, wherein the processor implements the following operations when executing the executable program instructions:

Based on the operation information input by the user, the position of the virtual element in the virtual scene is moved in real time. If the virtual element moves to the area corresponding to the virtual element in the special effect of the unfolded structure, the virtual element is fixedly displayed in the virtual scene. area.
The apparatus of claim 23, wherein the processor implements the following operations when executing the executable program instructions:

obtaining the relative pose of the camera and the virtual scene;

The selected virtual element is determined based on the operation information, and the selected virtual element is rendered at a second preset relative position of the camera in the virtual scene.
The apparatus of claim 24, wherein the processor implements the following operations when executing the executable program instructions:

The position of the selected virtual element in the virtual scene is moved based on the pose change of the camera in the virtual scene.
The apparatus of claim 24, wherein the processor implements the following operations when executing the executable program instructions:

Moves the position of the selected virtual element in the virtual scene based on the position selected by the user in the scene image.
The apparatus of claim 24, wherein the processor implements the following operations when executing the executable program instructions:

The output displays brief information about the selected virtual element.
The apparatus of claim 24, wherein the processor implements the following operations when executing the executable program instructions:

The posture of the selected virtual element at the second preset relative position of the camera is changed based on the operation information.
The apparatus of claim 24, wherein the processor implements the following operations when executing the executable program instructions:

The region identifier corresponding to the selected virtual element is rendered in the expanded structure special effect.
The apparatus of claim 23, wherein the processor implements the following operations when executing the executable program instructions:

If all the virtual elements are moved to the area corresponding to the virtual elements in the expanded structure effect, the expanded structure effect is rendered as the first virtual model.
The apparatus according to claim 30, wherein the apparatus is installed in an electronic device, and the processor implements the following operations when executing the executable program instructions:

A control component is rendered on the display screen of the electronic device for the user to control the movement of the first virtual model.
The apparatus according to claim 31, wherein the first virtual model moves based on a plane of a virtual scene; and the plane of the virtual scene is determined based on plane information of a real scene.
An electronic device, comprising:

camera, for capturing images;

processor;

memory for storing processor executable program instructions;

The processor implements the following operations when executing the executable program instructions:

Real-time acquisition of scene images of real scenes through cameras;

In the pre-built virtual scene, a virtual element for assembling a first virtual model corresponding to a product is displayed at a position corresponding to a specified position in the real scene, and the first virtual element is displayed at a first preset relative position of the virtual element. 1. The unfolding structure special effect of virtual model;

The virtual scene is rendered in the scene image, the display content of the virtual scene is adjusted based on the obtained operation information input by the user, and the adjusted virtual scene is re-rendered in the scene image.
The electronic device according to claim 33, wherein the processor implements the following operations when executing the executable program instructions:

Detect plane information in real scenes;

A plane in the virtual scene is determined based on the plane information.
The electronic device according to claim 33, wherein the processor implements the following operations when executing the executable program instructions:

The position selected by the user in the scene image is acquired, and the position in the real scene corresponding to the position selected by the user in the scene image is determined as the designated position.
The electronic device according to claim 35, wherein the processor implements the following operations when executing the executable program instructions:

Renders a prompt effect in the scene image that prompts the user to select a location in the scene image.
The electronic device according to claim 36, wherein the processor implements the following operations when executing the executable program instructions:

Stop rendering the prompt special effect in the scene image, and display a second virtual model in the pre-built virtual scene at a position corresponding to the specified position in the real scene, and render the virtual scene in the scene image;

The processor also implements the following operations when executing the executable program instructions:

When the user's selection operation on the second virtual model is received, the second virtual model is rendered in the pre-built virtual scene at a position corresponding to the specified position in the real scene, for assembling into the corresponding product of the product. A virtual element of the first virtual model.
The electronic device according to claim 33, wherein the processor implements the following operations when executing the executable program instructions:

A reference coordinate system is established at the position corresponding to the specified position in the real scene in the pre-built virtual scene;

The virtual scene is rendered in the scene image based on the transformation relationship between the reference coordinate system and the image coordinate system of the scene image.
The electronic device according to claim 33, wherein the processor implements the following operations when executing the executable program instructions:

The position of the virtual element in the virtual scene is moved in real time based on the operation information input by the user. If the virtual element moves to the area corresponding to the virtual element in the special effect of the unfolded structure, the virtual element is fixedly displayed in the virtual scene. area.
The electronic device according to claim 39, wherein the processor implements the following operations when executing the executable program instructions:

obtaining the relative pose of the camera and the virtual scene;

The selected virtual element is determined based on the operation information, and the selected virtual element is rendered at a second preset relative position of the camera in the virtual scene.
The electronic device according to claim 40, wherein the processor implements the following operations when executing the executable program instructions:

The position of the selected virtual element in the virtual scene is moved based on the pose change of the camera in the virtual scene.
The electronic device according to claim 40, wherein the processor implements the following operations when executing the executable program instructions:

Moves the position of the selected virtual element in the virtual scene based on the position selected by the user in the scene image.
The electronic device according to claim 40, wherein the processor implements the following operations when executing the executable program instructions:

The output displays brief information about the selected virtual element.
The electronic device according to claim 40, wherein the processor implements the following operations when executing the executable program instructions:

The posture of the selected virtual element at the second preset relative position of the camera is changed based on the operation information.
The electronic device according to claim 40, wherein the processor implements the following operations when executing the executable program instructions:

The region identifier corresponding to the selected virtual element is rendered in the expanded structure special effect.
The electronic device according to claim 39, wherein the processor implements the following operations when executing the executable program instructions:

If all the virtual elements are moved to the area corresponding to the virtual elements in the expanded structure effect, the expanded structure effect is rendered as the first virtual model.
The electronic device according to claim 46, wherein the electronic device further comprises a display screen, and the processor implements the following operations when executing the executable program instructions:

A control component is rendered on the display screen of the electronic device for the user to control the movement of the first virtual model.
The electronic device according to claim 47, wherein the first virtual model moves based on a plane of a virtual scene; and the plane of the virtual scene is determined based on plane information of a real scene.