WO2020168792A1

WO2020168792A1 - Augmented reality display method and apparatus, electronic device, and storage medium

Info

Publication number: WO2020168792A1
Application number: PCT/CN2019/125029
Authority: WO
Inventors: 程侠宽; 李一山; 周熠; 张岩
Original assignee: 北京三快在线科技有限公司
Priority date: 2019-02-18
Filing date: 2019-12-13
Publication date: 2020-08-27
Also published as: CN109903129A

Abstract

An augmented reality display method and apparatus, an electronic device, and a computer-readable storage medium. The method comprises: obtaining entity information in a collected real scene image (S101); for a three-dimensional model of a virtual article, extracting size information of the three-dimensional model (S102); determining display parameters of the three-dimensional model according to the entity information and the size information (S103); and displaying a three-dimensional model image of the virtual article in the real scene image on the basis of the display parameters (S104). The display parameters of the three-dimensional model of the virtual article are determined by means of an entity in the real scene image, so that a user can accurately find out an adaptation condition between the virtual article and a real scene, and more real experience is brought to the user, thereby increasing the purchase rate, and reducing the return rate; furthermore, human-computer interaction by means of gesture actions increases interestingness for online shopping of the user, and also enables the user to have immersive experience; the shopping mode is more flexible and accurate, and the conversion rate is increased.

Description

Augmented reality display method and device, electronic equipment, and storage medium

This disclosure claims the priority of a Chinese patent application filed on February 18, 2019 with the application number 201910120539.3 and the invention title "Augmented Reality Display Method and Device, Electronic Equipment, Storage Medium", the entire content of which is incorporated herein by reference Open.

Technical field

The present disclosure relates to the field of augmented reality (Augmented Reality, AR), and in particular to an augmented reality display method, augmented reality display device, electronic equipment, and computer-readable storage medium.

Background technique

With the development of technology, e-commerce is slowly approaching people's lives. With the promotion and popularization of online shopping, a new shopping experience has been brought to users' consumption. In the traditional online shopping mode, consumers view products in the form of pictures, but the product information that consumers can obtain is greatly restricted. In this case, an alternative shopping method is needed to provide consumers with the required information, so that consumers can intuitively understand product-related information and experience product functions anytime, anywhere.

It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art.

Summary of the invention

The purpose of the present disclosure is to provide an augmented reality display method, augmented reality display device, electronic equipment, and computer readable storage medium, thereby at least to some extent overcome the inability to associate product images with scenes and consumption due to limitations of related technologies Problems such as inability to interact with the product.

Other characteristics and advantages of the present disclosure will become apparent through the following detailed description, or partly learned through the practice of the present disclosure.

According to an aspect of the present disclosure, an augmented reality display method is provided, the method includes: acquiring entity information in a collected real scene image; extracting size information of the three-dimensional model for a three-dimensional model of a virtual item; The entity information and the size information determine the display parameters of the three-dimensional model; based on the display parameters, the three-dimensional model image of the virtual item is displayed in the real scene image.

In an exemplary embodiment of the present disclosure, the acquiring entity information in the collected real scene image includes: identifying at least one entity in the real scene image; acquiring boundary information and location information of the entity .

In an exemplary embodiment of the present disclosure, the determining the display parameters of the three-dimensional model includes: determining the display size and zoom ratio of the three-dimensional model; and determining the display position of the three-dimensional model.

In an exemplary embodiment of the present disclosure, the augmented reality display method further includes: in response to an interactive instruction for the virtual item, displaying the processed three-dimensional image of the virtual item in the real scene image. Model image.

In an exemplary embodiment of the present disclosure, the displaying the processed three-dimensional model image of the virtual item in the real scene image includes: processing the three-dimensional model of the virtual item to determine After processing the display parameters of the three-dimensional model of the virtual item, the processing includes at least one of the following: color processing, position processing, display size processing, and scaling processing; based on the processed three-dimensional model of the virtual item Display parameters, displaying the processed three-dimensional model image of the virtual item in the real scene image.

In an exemplary embodiment of the present disclosure, the augmented reality display method further includes: generating an interaction instruction for the virtual item in response to a user's gesture action.

In an exemplary embodiment of the present disclosure, the generating an interactive instruction for the virtual item for the user's gesture action includes: when it is determined that the gesture action is a first gesture, generating an interactive instruction for moving the virtual item A first interaction instruction of the item; when it is determined that the gesture action is a second gesture, a second interaction instruction for zooming the virtual item is generated.

In an exemplary embodiment of the present disclosure, the three-dimensional model of the virtual item is generated by the following steps: acquiring video data of the virtual item; extracting a preset number of static images from the video data; The static image generates a three-dimensional model of the virtual item.

According to one aspect of the present disclosure, an augmented reality display device is provided, the device includes: an information acquisition module configured to acquire entity information in a collected real scene image; an information extraction module configured to target virtual items The three-dimensional model extracts the size information of the three-dimensional model; the parameter determination module is configured to determine the display parameters of the three-dimensional model according to the entity information and the size information; the image display module is configured based on the display parameters , Displaying the three-dimensional model image of the virtual item in the real scene image.

According to one aspect of the present disclosure, there is provided an electronic device including: a processor and a memory; wherein a computer readable instruction is stored on the memory, and the computer readable instruction is executed by the processor to implement any of the foregoing exemplary implementations Examples of augmented reality display methods.

According to one aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the augmented reality display method in any of the foregoing exemplary embodiments.

The exemplary embodiments of the present disclosure have the following beneficial effects:

In the method and device provided by the exemplary embodiments of the present disclosure, the display parameters of the three-dimensional model of the virtual item are determined by the entity in the real scene image, so that the user can see exactly how the virtual item fits in the real scene. Give users a more realistic experience, increase the purchase rate, and reduce the return rate caused by the inconsistency of the purchased items with the real usage; further, the human-computer interaction through gestures not only adds fun to the user's online shopping , So that users have a sense of immersion, shopping methods more flexible and accurate, and improve conversion rate.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure.

Description of the drawings

The drawings herein are incorporated into the specification and constitute a part of the specification, show embodiments in accordance with the disclosure, and together with the specification are used to explain the principle of the disclosure. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

Fig. 1 schematically shows a flowchart of an augmented reality display method in an exemplary embodiment of the present disclosure;

Fig. 2 schematically shows a flowchart of partial steps of an augmented reality display method in an embodiment of the present disclosure;

FIG. 3 schematically shows a flowchart of partial steps of an augmented reality display method in an embodiment of the present disclosure;

FIG. 4 schematically shows a flowchart of partial steps of an augmented reality display method in an embodiment of the present disclosure;

FIG. 5 schematically shows a flowchart of partial steps of an augmented reality display method in an embodiment of the present disclosure;

FIG. 6 schematically shows a flowchart of steps of an augmented reality display method in an embodiment of the present disclosure;

FIG. 7 schematically shows a flowchart of steps of an augmented reality display method in an embodiment of the present disclosure;

FIG. 8 schematically shows a schematic diagram of an implementation scenario of an augmented reality display method in an exemplary embodiment of the present disclosure;

FIG. 9 schematically shows a schematic structural diagram of an augmented reality display device in an exemplary embodiment of the present disclosure;

FIG. 10 schematically shows an electronic device for implementing an augmented reality display method in an exemplary embodiment of the present disclosure;

FIG. 11 schematically shows a computer-readable storage medium for implementing an augmented reality display method in an exemplary embodiment of the present disclosure.

detailed description

Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in a variety of forms, and should not be construed as being limited to the examples set forth herein; on the contrary, the provision of these embodiments makes the present disclosure more comprehensive and complete, and fully comprehends the concept of the exemplary embodiments. To convey to those skilled in the art. The described features, structures or characteristics may be combined in one or more embodiments in any suitable way.

In addition, the drawings are only schematic illustrations of the present disclosure, and are not necessarily drawn to scale. The same reference numerals in the figures denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

It should be noted that in the present disclosure, the terms "include", "configured with", and "provided in" are used to mean open-ended inclusion, and mean in addition to the listed elements/components/etc. There may also be other elements/components/etc.; the terms "first", "second", etc. are used only as marks, and are not limitations on the number or order of their objects.

In the exemplary embodiment of the present disclosure, an augmented reality display method is first provided, which is applied to a terminal device. For example, the terminal device may be a computer or a notebook equipped with a camera, and multifunctional glasses with augmented reality display. It can also be a device with a smart terminal as the collection and processing side and a home TV with a communication link as the display side, and it can also be any other electronic terminal device with an augmented reality display function.

Referring to FIG. 1, the augmented reality display method may mainly include the following steps:

Step S101. Obtain entity information in the collected real scene image.

In practical applications, the camera device of the smart terminal can be used to collect real scene images. The camera device may include one camera or multiple cameras located at different angles. When the camera device only includes one camera, it can collect 360°, and collect a scene every 15°. The collection distance of the scene can be set to five types: 1 meter, 2 meters, 3 meters, 4 meters and 5 meters. Distance, the collection environment can be set to two scenes: indoor during the day and indoor at night; correspondingly, when the camera device includes multiple cameras, only different collection distances and collection environments can be set.

For different virtual items, it is necessary to select the most suitable entity as a reference in the collected real scene images. For the real scene image, the entity information needs to be obtained. The entity in the entity information may be an object contained in a real scene image, and the number of objects is not limited. For different entities, due to their different environments, they refer to different objects. For example, if the real scene is at home, the entity can be a coffee table, chair, sofa, table, etc.; when the real scene is a vehicle, the entity can be Seats, display screens, windows, etc.; when the real scene is an office, the entity can be a drinking fountain, desk, computer, telephone, etc.; when the real scene is a shopping mall, the entity can be a model, a full-length mirror, etc. Then, entity information refers to information related to the entity, such as size information, location information, color information, shape information, environmental information, etc., where environmental information is related to the environment in which the entity is located, such as the distance between the entity and the wall, The brightness of the environment, the distance from other entities in the environment, etc.

Step S102: Extract the size information of the three-dimensional model for the three-dimensional model of the virtual item.

Through step S101, the entity information in the collected real scene image can be obtained. In this step, the size information of the three-dimensional model of the virtual item can be obtained. Generally, virtual items are presented in the form of 3D models. Wherein, the virtual item may be an item in an online store, such as furniture, clothing, equipment, musical instruments, accessories, etc. The item image can be obtained by entering the name on the display page to obtain the provided item image, or by using a camera tool to capture the item to obtain the item image. Use the acquired image of the item to generate a three-dimensional model of the corresponding virtual item, and obtain size information of the three-dimensional model. For example, when the virtual item is a table, the characteristic information of the three-dimensional model of the table may be its length, width, height, and other dimensions. Different size information can be obtained for different virtual items to facilitate subsequent operations.

Step S103: Determine the display parameters of the three-dimensional model according to the entity information and the size information.

The entity information in the real scene image is obtained in step S101, and the size information of the three-dimensional model of the virtual item is obtained in step S102. In this step, the three-dimensional model of the online virtual item can be displayed in the real scene image through the entity information And size information determines the display parameters of the three-dimensional model. If the entity in the real scene image is a chair, and the entity information of the chair has been obtained, that is, its size, shape, position, and color information, when the virtual item is a table, it can be based on the size information and the obtained entity information The size relationship of the table determines the size ratio relationship between the two to determine the display size of the table, and the display parameters such as the display position and adaptation color of the table are determined according to other entity information of the chair.

Step S104. Based on the display parameters, display the three-dimensional model image of the virtual item in the real scene image.

In step S103, the display parameters of the three-dimensional model of the virtual item have been determined. In this step, the three-dimensional model image of the virtual item can be displayed in the real scene image. In order to intervene the three-dimensional model image of the virtual item in the real scene image captured by the camera device, so that the user can feel the existence of the virtual item in the real scene, the relevant parameters of the camera device need to be set, and the virtual item’s The 3D model image is used as a reference, and the 3D model image and the real scene image are superimposed. For example, the virtual item surrounds the entity in the real scene, or covers the entity, or is placed next to the entity.

The display of the three-dimensional model image of the virtual item in the real scene image can be on the glasses lens of the augmented reality display, or on the screen of the smart terminal, such as the display screen of a mobile phone, a computer, and a home TV.

According to the augmented reality display method in this exemplary embodiment, the relevant display information of the three-dimensional model of the virtual item that needs to be displayed is determined by the entity in the real scene, thereby determining the most realistic virtual item size information; further, the virtual item The product is displayed in the real scene, which is of great reference value for users to determine whether they meet expectations and evaluate whether to purchase, improve the success rate of online purchases, and reduce the occurrence of returns and exchanges due to the inability to grasp product information, saving Cost and resources.

On the basis of the above embodiment, as shown in FIG. 2, acquiring entity information in a real scene image includes the following steps:

Step S201: Identify at least one entity in the real scene image.

In step S101, a real scene image can be collected, and in this step, entities in the real scene image can be identified. The entity recognition in the real scene image can firstly be the recognition of the entire real scene image, or it can be the recognition of a specific area in the real scene image. Before recognizing the real scene image, we should first realize the preprocessing of the real scene image. The preprocessing methods can include: downsampling, binarization, filtering, etc. Among them, downsampling can be used to improve the performance of the real scene image, so that the real scene image meets the size of the area to be displayed; binarization can make the selected feature points not only depend on the color, but also more clearly; the use of Gaussian filtering can reduce noise and make The acquired image of the real scene is not affected by scratches, dust, etc.

Recognize entities in real scene images. There is no limit to the number of recognized entities. It can be one or multiple. The more recognized entities, the greater the display size and/or scaling of the 3D model. The ratio and positioning of the display position are more accurate. The way of recognizing an entity may be to recognize a specific color or a specific shape, as long as the required entity can be recognized, and this exemplary embodiment does not limit the recognition mode.

Step S202: Obtain boundary information and location information of the entity.

In step S201, the entity in the real scene image has been identified. In this step, relevant information of the identified entity can be obtained, and the relevant information may be boundary information and location information of the entity. The boundary information includes information corresponding to the boundary of the space where the entity is located, and the location information includes information corresponding to the location of the entity in the real scene image. For example, the boundary information can be the length, width, and height of the entity. For example, to obtain the location information of an entity, a three-dimensional coordinate system of the entity in the real scene image can be established, and the camera coordinate system of the camera device that captures the real scene image can be established. According to the mutual transformation between the three-dimensional coordinate system and the camera coordinate system, Get the location information of the entity. The boundary information of the entity can be determined by image recognition algorithms, such as Sobel edge detection algorithm, Laplacian operator, etc.

This embodiment can identify entities in the real scene image and determine its related information, and can accurately grasp the entity information in the real environment, which has very important reference significance for determining the display parameters of virtual items.

On the basis of the above embodiment, as shown in Fig. 3, obtaining a three-dimensional model of a virtual item includes the following steps:

Step S301: Obtain video data of the virtual item.

Merchants can use shooting terminals to shoot videos of virtual items and obtain video data. Among them, the shooting terminal may include a surveillance camera, a front camera of a smart terminal, a rear camera of a smart terminal, etc., and may also include a processing terminal that processes video data after acquiring it. The collected video data can be stored in the database and server as xml, json and other data.

Step S302. Extract a preset number of static images from the video data.

In step S301, the video data of the virtual item has been collected. In this step, static images can be extracted from the video data. By setting the frame number interval for extracting the static image, and the starting point of the frame number interval can be the frame number when the static image was acquired last time, the purpose of extracting the static image with the preset number of frames is achieved. The frame number interval should not be set too small, otherwise the virtual item has not changed and the static image is extracted, which will cause the number of extracted static images to be too much, resulting in waste of resources; the frame number interval should not be set too large, otherwise the number of static images extracted will be too large If it is not enough, it will lead to insufficient extraction of effective information and reduce user experience. Generally, in order to make the three-dimensional model of the virtual item present a more realistic three-dimensional sense, the static images of the preset number of frames can include the front view, back view, left view, right view, top view and bottom view of the virtual item. . The method of extracting static images may be a coding method using the image compression standard of JPEG (Joint Photographic Experts Group, Joint Photographic Experts Group).

Step S303: Use the static image to generate a three-dimensional model of the virtual item.

In step S302, several static images of the virtual item can be extracted. In this step, the static images will be used to generate a three-dimensional model of the virtual item. The technology of building a three-dimensional model through static images can use smart devices to perform graphic processing and three-dimensional calculations on the collected static images, thereby automatically generating a three-dimensional model of virtual objects.

This embodiment describes the generation of a three-dimensional model of a virtual item. The static image is obtained through video data to construct the required three-dimensional model. The generation method is simple and the operation is convenient. The three-dimensional model constructed from the actual static image is more realistic and the product is improved. The look and feel improves the user’s shopping experience.

On the basis of the above embodiments, as shown in Fig. 4, determining the display parameters of the three-dimensional model includes the following steps:

Step S401: Determine the display size and zoom ratio of the three-dimensional model.

According to the entity in the real scene image, the display parameters of the 3D model can be determined so that it can not only reflect the real information of the virtual item, but also reflect its most appropriate and suitable size in the environment. The user can intuitively and clearly Know whether the virtual item matches the user's real use environment. For example, if the three-dimensional model is a table, the display size and zoom ratio of the table in the appropriate placement area can be determined according to the size and ratio of the stool or sofa in the actual scene image. The display size refers to the actual scene The size of the 3D model is displayed in the image, such as the height, length, and width of the table. The zoom ratio refers to the proportion of the 3D model that needs to be reduced or enlarged according to the actual scene image, such as reduced to 80% and enlarged to 120%.

Step S402: Determine the display position of the three-dimensional model.

In step S401, the display size and zoom ratio of the three-dimensional model can be determined. In this step, the display position of the three-dimensional model can be determined. The display position refers to the position of the three-dimensional model in the real scene image. The display position of the 3D model should be the most suitable position such as the placement position and the use position in the real scene image. For example, when the virtual item is a table, its display position should be directly in front of a stool or sofa, and keep an appropriate distance.

Corresponding to the entity information in the real scene image, this embodiment obtains the size information of the three-dimensional model, and uses the proportional relationship between the two to determine the display size and zoom ratio corresponding to the best display effect of the three-dimensional model, as well as the display position, so as to avoid virtual Objects are too big or too small, and the abrupt feeling caused by poor placement.

On the basis of the above embodiment, the augmented reality display method further includes: in response to an interactive instruction for the virtual item, displaying a processed three-dimensional model image of the virtual item in the real scene image.

When the three-dimensional model of the virtual item is placed in the real scene image, it is impossible to determine that the default color at the beginning is the most suitable color, nor can it be determined that the initial display surface is the side with the best display effect. The 3D model image is adjusted. When the user implements a gesture action, it is equivalent to sending an interactive instruction for the virtual item, which can complete actions such as zooming, rotating, and switching the three-dimensional model. When the user has selected the best display effect of the virtual item, the interaction process with the three-dimensional model image of the virtual item in the real scene image is completed.

On the basis of the above embodiments, in response to the interactive instruction for the virtual item, displaying the processed three-dimensional model image of the virtual item in the real scene image includes: processing the three-dimensional model of the virtual item and determining the processed three-dimensional model image Display parameters of the three-dimensional model of the virtual item; based on the display parameters of the processed three-dimensional model of the virtual item, display the processed three-dimensional model image of the virtual item in the real scene image; wherein the above processing includes at least one of the following: Color processing, position processing, display size processing, and scaling processing. Among them, color processing refers to adjusting the color of the 3D model, position processing refers to adjusting the display position of the 3D model, and display size processing refers to adjusting the display size of the 3D model, zooming Scaling refers to adjusting the zoom ratio of the 3D model.

In this embodiment, through an interactive instruction, the user can interact with the 3D model of the virtual item in the real scene image, and display the 3D model image after the interaction, which can ensure the user's sense of real substitution and the personalization of different users. Optionally, the virtual items can be placed in the real scene in the best display state, ensuring the best display effect of online shopping.

On the basis of the above embodiment, the augmented reality display method further includes: generating an interactive instruction for the virtual item in response to the user's gesture action.

When acquiring the user's gesture action, the user's hand dynamic image can be collected first. The dynamic image of the user's hand is acquired through the camera device of the smart terminal or the depth camera on the augmented reality device, and the dynamic image of the hand is analyzed. Acquiring real-time hand information of the user's hand may include valid gesture actions and invalid gesture actions, and valid gesture actions refer to preset gesture actions. At least one preset gesture action should be set in the augmented reality device, for example, thumb and index finger closing, index finger rotation, index finger flipping and other gesture actions, but it is not limited to using specific fingers or specific actions. A preset gesture action is set with corresponding operations, such as zooming, rotating, and switching virtual items. When the user's gesture action is obtained, the gesture action can be matched with the preset gesture action. If the matching is successful, the gesture action can perform the corresponding operation. If the matching is unsuccessful, the gesture action is regarded as an invalid gesture action. When the user interacts with the three-dimensional model of the virtual item and needs to adjust it, such as zooming, rotating, switching, etc., he can send an interactive instruction to perform related operations through the executed gesture operation.

This embodiment uses the user’s gestures to issue an interactive instruction to the virtual item to perform operations on the virtual item. On the one hand, it can improve the fluency and accuracy of the user’s shopping, and facilitate the user to adjust the display state of the three-dimensional model to achieve the desired value. On the other hand, it can break the barriers that cannot interact and communicate between users and products in traditional online shopping methods, and improve the immersion of users' online shopping experience.

On the basis of the above embodiment, as shown in FIG. 5, generating an interactive instruction for the virtual item in response to a user's gesture action may include the following steps:

Step S501. When it is determined that the gesture action is the first gesture, a first interaction instruction for moving the virtual item is generated.

For the user's gesture action that has been acquired, this step lists scenarios when it is determined that the acquired gesture action is the first gesture. When the acquired gesture action is a preset action and is the first gesture, the first gesture may be the index finger flip. When the gesture action occurs, it is equivalent to triggering the first interactive instruction, which corresponds to moving the virtual The manipulation of items.

Step S502. When it is determined that the gesture action is the second gesture, a second interaction instruction for zooming the virtual item is generated.

In step S501, the user's gesture action has been acquired, and this step lists scenarios when it is determined that the acquired gesture action is the second gesture. When the acquired gesture action is a preset action and is a second gesture, the second gesture may be the closing of the thumb and index finger. As long as the gesture action occurs, it is equivalent to triggering the second interactive instruction, which corresponds to the operation of zooming the virtual item. Among them, the second interactive instruction can include two situations: when the distance between the thumb and the index finger changes from small to large, the virtual item can be reduced; when the distance between the thumb and index finger changes from large to small, it can be It is to enlarge the virtual item. Or, the second interactive instruction can include two situations: when the distance between the thumb and index finger is changed from large to small, the virtual item can be reduced; when the distance between the thumb and index finger is changed from large to small, it can be It is to enlarge the virtual item.

This embodiment lists the adjustment of the display state of the virtual items corresponding to the two gesture actions, and explains the user's interaction mode, so that the user can operate the virtual items more accurately when shopping, and the virtual items are more realistic and the shopping mode is better. For flexibility.

It should be noted that although the above exemplary embodiments describe the steps of the method in the present disclosure in a specific order, this does not require or imply that these steps must be performed in the specific order, or that all steps must be performed. In order to achieve the desired result. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and/or one step may be decomposed into multiple steps for execution, etc.

On the basis of the above embodiments, FIG. 6 schematically shows an implementation environment of the augmented reality display method of the present disclosure, and the implementation environment includes: a server 10 and a terminal device 20.

The server 10 refers to a device with computing capability and storage capability. Optionally, the server 10 may be a server, a server cluster composed of multiple servers, or a cloud computing center. In the embodiment of the present disclosure, the merchant 30 can shoot the product. Optionally, the merchant 30 can shoot the product in 360 degrees, and then the merchant 30 uploads the captured video data to the server 10, and the server 10 receives the video data Then, the video data is processed to obtain a three-dimensional model.

The terminal device 20 refers to a device with data acquisition capabilities and image processing capabilities. Optionally, the terminal device 20 includes electronic terminals such as mobile phones, tablet computers, multimedia playback devices, and wearable devices. In the embodiment of the present disclosure, the user 40 can view the merchandise of the merchant 30 in the terminal device 20. When the user 40 determines to display a merchandise, the terminal device 20 receives the request of the user 40 and obtains the product from the server 10 according to the request. The three-dimensional model corresponding to the product. The terminal device 20 may collect a real scene image, and obtain boundary information and position information of entities in the real scene image, and then synthesize the real scene image and the above-mentioned three-dimensional model. The user 40 can also adjust the three-dimensional model in the terminal device 20, such as zooming, selecting a position, etc. The terminal device 20 adjusts the three-dimensional model in the real scene image according to the operation of the user 40, and displays the adjusted composite image.

In the embodiment of the present disclosure, the terminal device 20 and the server 10 may communicate through a network. Optionally, the network may be a wired network or a wireless network.

Below, a specific example is used to introduce and explain the method of the present disclosure.

As shown in Figure 7, assuming that the virtual item is a virtual desk lamp, the generation process of the three-dimensional model of the virtual desk lamp is as follows:

Step 910, the merchant photographs the physical desk lamp; the merchant refers to the seller who sells the physical desk lamp, and the physical desk lamp refers to the desk lamp displayed in the real physical world;

Step 920: The merchant uploads the video data obtained by shooting to the server; where the video data includes a virtual desk lamp, which is opposite to the above-mentioned physical desk lamp and refers to the desk lamp in the video data processed by the server or other equipment;

In step 930, the server generates a three-dimensional model of the virtual table lamp according to the video data; after the server obtains the video data, it can extract static images of a preset number of frames from the video data, and then generates a three-dimensional model of the virtual table lamp according to the static images, for example, The three-dimensional model of the virtual desk lamp is shown as 1010 in Figure 8;

Assuming that the real scene image is a room that includes a desk, the user can experience the display effect of the above-mentioned virtual desk lamp on the desk of the room through the terminal device. The display method of the virtual desk lamp on the desk is as follows:

Step 940: The user determines to display the virtual table lamp; the user can select the virtual table lamp from the merchandise sold by the merchant, and determine to experience the display effect of the virtual table lamp in the real scene image in the terminal device;

Step 950, the terminal device obtains the three-dimensional model of the virtual table lamp; the terminal device obtains the three-dimensional model of the virtual table lamp from the server in response to the user's selection;

Step 960: The terminal device recognizes at least one entity in the real scene image, and determines the boundary information and position information of each entity; the real scene image may be an image taken by the user using the terminal device, and the real scene image may include at least one entity, For example, as shown in FIG. 8, the real scene image is a room 1020, and the real scene image includes a desk 1030. The desk 1030 is an entity, and the terminal device can determine the boundary information and location information of the desk;

Step 970: The terminal device determines the display size, zoom ratio and display position of the 3D model according to the boundary information and location information of the entity, and displays the 3D model in the real scene image; the terminal device according to the 3D model of the virtual desk lamp and the real scene image The boundary information and position information of the entity in the middle can calculate the display size, zoom ratio and display position of the 3D model. The values of these parameters are the initial default values calculated by the terminal, and then the terminal displays the 3D model in the real scene image according to these parameters. For example, as shown in FIG. 8, the terminal device displays the three-dimensional model 1010 in the real scene image 1020 according to the desk 1030 and the three-dimensional model 1010 in the real scene image 1020;

In step 980, the user adjusts the 3D model in the real scene image; the display position of the 3D model calculated by the terminal device may not be the best display, and the user can compare the 3D model in the real scene image according to their individual needs Make adjustments

In step 990, the terminal device displays the adjusted three-dimensional model in the real scene image according to the user's adjustment; for example, as shown in FIG. 8, assuming that the user's adjustment is to enlarge the three-dimensional model 1010 and move it to the left, the terminal device The adjustment of the user displays the adjusted three-dimensional model 1010 in the real scene image 1020.

In addition, in an exemplary embodiment of the present disclosure, an augmented reality display device is also provided. Referring to FIG. 9, the augmented reality display device 600 may include: an information acquisition module 601, an information extraction module 602, a parameter determination module 603, and an image display module 604. among them:

The information acquisition module 601 is configured to acquire entity information in the collected real scene images; the information extraction module 602 is configured to extract size information of the three-dimensional model for the three-dimensional model of the virtual item; the parameter determination module 603 is configured to Configured to determine the display parameters of the three-dimensional model according to the entity information and the size information; the image display module 604 is configured to display the three-dimensional model of the virtual item in the real scene image based on the display parameters image.

The specific details of the above-mentioned augmented reality display device have been described in detail in the corresponding augmented reality display method, and therefore will not be repeated here.

It should be noted that although several modules or units of the augmented reality display device 600 are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

The electronic device 700 according to this embodiment of the present invention will be described below with reference to FIG. 10. The electronic device 700 shown in FIG. 10 is only an example, and should not bring any limitation to the function and application scope of the embodiment of the present invention.

As shown in FIG. 10, the electronic device 700 is represented in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: the aforementioned at least one processing unit 710, the aforementioned at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

Wherein, the storage unit stores program code, and the program code can be executed by the processing unit 710, so that the processing unit 710 executes the various exemplary methods described in the "exemplary method" section of this specification. Example steps.

The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 721 and/or a cache storage unit 722, and may further include a read-only storage unit (ROM) 723.

The storage unit 720 may also include a program/utility tool 724 having a set of (at least one) program module 725. Such program module 725 includes but is not limited to: an operating system, one or more application programs, other program modules, and program data, Each of these examples or some combination may include the implementation of a network environment.

The bus 730 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local area using any bus structure among multiple bus structures. bus.

The electronic device 700 can also communicate with one or more external devices 900 (such as keyboards, pointing devices, Bluetooth devices, etc.), and can also communicate with one or more devices that enable users to interact with the electronic device 700, and/or communicate with Any device (such as a router, modem, etc.) that enables the electronic device 700 to communicate with one or more other computing devices. This communication can be performed through an input/output (Input/Output, I/O) interface 750. In addition, the electronic device 700 may also communicate with one or more networks (for example, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through the network adapter 760. As shown in the figure, the network adapter 740 communicates with other modules of the electronic device 700 through the bus 730. It should be understood that although not shown in the figure, other hardware and/or software modules can be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.

Through the description of the above embodiments, those skilled in the art can easily understand that the exemplary embodiments described here can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, and the software product can be stored in a non-volatile storage medium (which can be a CD-ROM (Compact Disc Read-Only Memory)) , U disk, mobile hard disk, etc.) or on the network, including several instructions to make a computing device (may be a personal computer, server, terminal device, or network device, etc.) execute the method according to the embodiments of the present disclosure.

In the exemplary embodiment of the present disclosure, there is also provided a computer-readable storage medium on which is stored a program product capable of implementing the above method in this specification. In some possible embodiments, various aspects of the present invention may also be implemented in the form of a program product, which includes program code, and when the program product runs on a terminal device, the program code is used to make the The terminal device executes the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of this specification.

As shown in FIG. 11, a program product 800 for implementing the above method according to an embodiment of the present invention is described. It may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be in a terminal device, For example, running on a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or device.

The program product can use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (Read Only Memory, ROM), Erasable Programmable Read-Only Memory (EPROM (Electrically Programmable Read-Only Memory) or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, Or any suitable combination of the above.

The computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than a readable storage medium, and the readable medium may send, propagate, or transmit a program for use by or in combination with the instruction execution system, apparatus, or device.

The program code contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF (Radio Frequency), etc., or any suitable combination of the foregoing.

The program code used to perform the operations of the present invention can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural styles. Programming language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computing device, partly on the user's device, executed as an independent software package, partly on the user's computing device and partly executed on the remote computing device, or entirely on the remote computing device or server Executed on. In the case of a remote computing device, the remote computing device can be connected to a user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using Internet service providers) Business to connect via the Internet).

Those skilled in the art will easily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The description and embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

Claims

An augmented reality display method, characterized in that it comprises:

Obtain entity information in the collected real scene images;

For the three-dimensional model of the virtual item, extract the size information of the three-dimensional model;

Determine the display parameters of the three-dimensional model according to the entity information and the size information;

Based on the display parameters, a three-dimensional model image of the virtual item is displayed in the real scene image.
The augmented reality display method according to claim 1, wherein the acquiring entity information in the collected real scene image comprises:

Identifying at least one entity in the real scene image;

Obtain boundary information and location information of the entity.
The augmented reality display method according to claim 1, wherein the determining the display parameters of the three-dimensional model comprises:

Determining the display size and zoom ratio of the three-dimensional model;

Determine the display position of the three-dimensional model.
The augmented reality display method according to claim 1, wherein the method further comprises:

In response to the interactive instruction for the virtual item, the processed three-dimensional model image of the virtual item is displayed in the real scene image.
The augmented reality display method according to claim 4, wherein the displaying the processed three-dimensional model image of the virtual item in the real scene image comprises:

Processing the three-dimensional model of the virtual item to determine display parameters of the processed three-dimensional model of the virtual item, the processing includes at least one of the following: color processing, position processing, display size processing, and scaling processing;

Based on the processed display parameters of the three-dimensional model of the virtual item, displaying the processed three-dimensional model image of the virtual item in the real scene image.
The augmented reality display method of claim 4, wherein the method further comprises:

An interactive instruction for the virtual item is generated for the user's gesture action.
The augmented reality display method according to claim 6, wherein said generating an interactive instruction for said virtual item for said gesture action of a user comprises:

When it is determined that the gesture action is the first gesture, generate a first interaction instruction for moving the virtual item;

When it is determined that the gesture action is a second gesture, a second interaction instruction for zooming the virtual item is generated.
The augmented reality display method according to any one of claims 1 to 7, wherein the three-dimensional model of the virtual item is generated by the following steps:

Acquiring video data of the virtual item;

Extracting a preset number of static images from the video data;

The static image is used to generate a three-dimensional model of the virtual item.
An augmented reality display device is characterized by comprising:

The information acquisition module is configured to acquire entity information in the collected real scene images;

An information extraction module configured to extract size information of the three-dimensional model of the virtual item;

A parameter determination module configured to determine the display parameters of the three-dimensional model according to the entity information and the size information;

The image display module is configured to display the three-dimensional model image of the virtual item in the real scene image based on the display parameter.
An electronic device, characterized in that it comprises:

Processor; and

A memory on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.
A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 8.