CN115599991A

CN115599991A - Method for providing commodity search result information and electronic equipment

Info

Publication number: CN115599991A
Application number: CN202211158070.0A
Authority: CN
Inventors: 刘邦哲; 罗阳; 曾超然; 王楠; 陈金珠
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2022-09-22
Filing date: 2022-09-22
Publication date: 2023-01-13

Abstract

The embodiment of the application discloses a method and electronic equipment for providing commodity search result information, wherein the method comprises the following steps: responding to a request for browsing commodity information in a Virtual Reality (VR) mode, and displaying a pre-generated virtual space scene based on VR technology, wherein the virtual space scene is generated in the following way: the method comprises the steps of carrying out live-action shooting on the display condition of the entity commodity in an entity space place, carrying out three-dimensional reconstruction on the space structure of the entity space place, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene; performing subject identification from a live-action shooting image frame corresponding to a current display visual angle in the virtual space scene; obtaining a commodity information search result related to the identified subject; and generating an information card to be displayed according to the commodity information search result, and mapping the information card to the virtual space scene for displaying. According to the embodiment of the application, richer commodity searching modes can be provided for the user.

Description

Method for providing commodity search result information and electronic equipment

Technical Field

The present application relates to the field of product search technologies, and in particular, to a method and an electronic device for providing product search result information.

Background

In the commodity information service system, the traditional commodity search service is carried out based on keywords, that is, a user inputs commodity keywords as search conditions, and then the system provides a search result page which accords with the search conditions. With the development of technologies such as mobile terminal devices, some commodity information service systems also provide a function of searching commodities based on pictures for users, so that the users can take pictures of interested articles when seeing the interested articles in real life, or store the pictures locally if the interested articles in the pictures are browsed on the internet. And then locally stored photos or pictures and the like can be uploaded to the server. Of course, the shooting can also be performed directly through the shooting entrance provided at the search control in the client page, and uploaded to the server, and so on. After receiving the photo or the picture, the server side can determine and return a commodity search result related to the main body on the photo by carrying out recognition, comparison and other processing on the main body in the picture. Furthermore, the user can search for a desired product from the search result list, and can perform operations such as online purchase.

That is, in the prior art, how to provide a richer commodity search method for a user by performing commodity search through either keyword input or picture uploading/taking becomes a technical problem to be solved by those skilled in the art.

Disclosure of Invention

The application provides a method and electronic equipment for providing commodity search result information, and richer commodity search modes can be provided for users.

The application provides the following scheme:

a method of providing merchandise search result information, comprising:

responding to a request for browsing commodity information in a Virtual Reality (VR) mode, and displaying a pre-generated virtual space scene based on VR technology, wherein the virtual space scene is generated in the following way: the method comprises the steps of carrying out live-action shooting on the display condition of the entity commodity in an entity space place, carrying out three-dimensional reconstruction on the space structure of the entity space place, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene;

performing subject identification from a live-action shooting image frame corresponding to a current display visual angle in the virtual space scene;

acquiring a commodity information search result related to the identified subject;

and generating an information card to be displayed according to the commodity information search result, and mapping the information card to the virtual space scene for displaying.

Wherein the live-action image is obtained by live-action shooting of the physical commodity display condition in the physical space place by the robot equipment in the physical space place.

Wherein, a plurality of different subspaces are distributed in the physical space place, and the physical commodities comprise physical commodities displayed in the subspaces;

the live-action shot image is obtained by live-action shooting of the robot equipment in the physical space place on the physical commodity display conditions in a plurality of different subspaces in the physical space place;

and when the three-dimensional reconstruction of the space structure is carried out, the three-dimensional reconstruction is carried out on the space distribution structure of a plurality of subspaces in the entity space site.

Wherein, still include:

providing, by the robotic device, an Augmented Reality (AR) mode-based shopping guide service for a user entering the physical space venue:

the shopping guide service provided by the robot equipment comprises: and acquiring a real-time image stream of the display condition of the entity commodity in the entity space place, identifying a subject from the real-time image stream, acquiring a commodity information search result related to the identified subject, generating an information card to be displayed according to the commodity information search result, and displaying the information card.

Wherein, when showing the information card through the robot apparatus, include:

and mapping the information card to the position of the main body in the corresponding image frame for displaying.

and displaying the information card in a holographic projection mode.

Wherein, when the mode through holographic projection is to the information card demonstrates, still include:

and displaying the 3D model of the commodity corresponding to the information card in a holographic projection mode.

Wherein the subject recognition from live-action captured image frames in the virtual space scene comprises:

during the process of dynamically changing the visual angle and/or performing forward and backward operations in the virtual space scene, performing subject identification from a plurality of live-action image frames, and acquiring the state change condition of a subject in the plurality of live-action image frames, wherein the state is related to the position of the subject in the live-action image frames and/or the area ratio of a subject image in a screen;

the generating of the information card to be displayed according to the commodity information search result includes:

and generating an information card to be displayed according to the commodity information search result and the state of the main body, wherein different main body states correspond to different information card types.

Reading the latest position of the identified main body according to a preset period in the process that the information card moves along with the position change of the main body; wherein the time interval is related to human eye persistence time and is greater than the time interval between each frame in the image stream;

and after the latest position is read, generating the animation which smoothly moves from the current position to the latest position for the information card which is displayed currently, keeping the position of the information card at the latest position unchanged, and generating new animation until a new latest position is read in the next period.

The relative position relation between the information card and the anchor point of the main body is various; the position of the anchor point is determined according to the center point of the main body;

the method further comprises the following steps:

and in the process that the information card moves along with the position change of the main body, judging whether the information card collides with the edge of the screen in the current relative position relation state, and if so, switching to other relative position relations.

If a plurality of main bodies are identified in the image frame, respectively acquiring corresponding commodity information search results, and respectively generating information cards of corresponding types according to respective states;

the method further comprises the following steps:

in the process that a plurality of information cards are mapped into the image frame and move along with the change of the positions of the corresponding main bodies, judging whether the plurality of information cards collide with each other or not;

if so, moving the information card display position corresponding to the lower-priority main body according to different main body state priorities corresponding to the collided information cards.

The information cards corresponding to different main body states are displayed on different layers, and the layers are arranged according to the priority of the main body states from high to low.

Wherein, still include:

in the process of mapping the information card to the position of the corresponding main body in the image frame for displaying, if the state of the main body changes, the type of the information card mapped to the virtual space scene is switched.

The buffer areas exist at the judgment parameter boundaries of different main body states, so that when the judgment parameters are changed in the buffer areas, the current main body state and the types of the corresponding information cards are kept unchanged.

The state of the main body comprises a display state, and the display state comprises a first display state and a second display state; the information card is used for displaying the content related to the commodity search result in the display state; the sizes and the content amounts of the information cards corresponding to the first display state and the second display state are different.

The state of the main body further comprises a discovery state, and the information card is used for displaying the identification result of the category to which the main body belongs in the discovery state.

Wherein the state of the subject further comprises an interactive state;

the method further comprises the following steps:

and after the main body enters the interaction state and keeps the threshold time, skipping to a commodity search result list page or displaying a detail page of the commodity in the currently displayed information card.

Wherein, in case the virtual space scene is presented by a VR or mixed reality MR glasses device, the method further comprises:

and when the VR or MR glasses detect that the user performs the operation of grabbing the information card and moving the information card to the target direction, adding the commodity corresponding to the information card into the batch settlement commodity set associated with the user.

A robot apparatus is provided, which is capable of,

the robot device is applied to a physical space place;

the robot equipment is used for providing shopping guide service based on an AR mode for a user entering the physical space place; the shopping guide service comprises: and acquiring a real-time image stream of the display condition of the entity commodity in the entity space place, identifying a subject from the real-time image stream, acquiring a commodity information search result related to the identified subject, generating an information card to be displayed according to the commodity information search result, and displaying the information card.

The robot equipment is further used for carrying out live-action shooting on the physical commodity display condition in the physical space place so as to carry out three-dimensional reconstruction on the space structure of the physical space place, and splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure to generate a virtual space scene;

in the process of VR display of the virtual space scene through terminal equipment associated with a user, subject identification is carried out from a live-action shooting image frame corresponding to a current display visual angle, a commodity information search result related to the identified subject is obtained, an information card to be displayed is generated according to the commodity information search result, and the information card is mapped to the virtual space scene for display.

An information display method, comprising:

displaying a pre-generated virtual space scene based on VR technology, the virtual space scene being generated in the following manner: performing live-action shooting on an entity in an entity space site, performing three-dimensional reconstruction on a space structure of the entity space site, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene;

acquiring an information search result related to the subject based on the identified subject image;

and generating an information card to be displayed according to the information search result, and mapping the information card to the virtual space scene for displaying.

An apparatus for providing commodity search result information, comprising:

the VR display unit is used for responding to a request for browsing commodity information through a virtual reality VR mode, displaying a pre-generated virtual space scene based on VR technology, and the virtual space scene is generated in the following mode: the method comprises the steps of carrying out live-action shooting on the display condition of the entity commodity in an entity space place, carrying out three-dimensional reconstruction on the space structure of the entity space place, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene;

the main body identification unit is used for carrying out main body identification from a live-action shooting image frame corresponding to a current display visual angle in the virtual space scene;

a product search result acquisition unit for acquiring a product information search result related to the identified subject;

and the information card display unit is used for generating an information card to be displayed according to the commodity information search result and mapping the information card to the virtual space scene for display.

An information presentation device comprising:

the VR display unit is used for displaying a pre-generated virtual space scene based on VR technology, and the virtual space scene is generated in the following mode: performing live-action shooting on an entity in an entity space place, performing three-dimensional reconstruction on a space structure of the entity space place, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene;

the subject identification unit is used for carrying out subject identification on a live-action shooting image frame corresponding to a current display visual angle in the virtual space scene;

a search unit configured to acquire an information search result related to the subject based on the recognized subject image;

and the information card display unit is used for generating an information card to be displayed according to the information search result, and the current display visual angle is correspondingly mapped to the virtual space scene for display.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method of any of the preceding claims.

An electronic device, comprising:

one or more processors; and

memory associated with the one or more processors for storing program instructions which, when read and executed by the one or more processors, perform the steps of the method of any of the preceding claims.

According to the specific embodiments provided by the application, the application discloses the following technical effects:

by the embodiment of the application, the real-scene shooting can be performed on the display condition of the entity commodity in the entity space place in advance, the three-dimensional reconstruction is performed on the space structure of the entity space place, the real-scene shot images are spliced and matched into the three-dimensional reconstructed virtual space structure, and the virtual space scene is generated. In this way, after a user initiates a request for browsing commodity information in a VR mode, a pre-generated virtual space scene can be displayed through VR technology, and subject recognition is performed from an image frame in the virtual space scene. Then, a commodity information search result related to the identified main body can be obtained, an information card to be displayed is generated according to the commodity information search result, and the information card is mapped to the position of the corresponding main body in the image frame for displaying. By the method, a user can obtain the experience of shopping mall online, the live-action shooting image frame contained in the virtual space scene can be used as the starting point of commodity searching with the picture, the commodity searching result is automatically obtained through main body identification and is mapped back to the virtual space scene in the form of the information card for displaying, so that the information in the virtual space scene created in the live-action shooting mode is richer, and meanwhile, the user can obtain brand-new commodity searching interaction experience.

In addition, in order to facilitate creation and update maintenance of the virtual space scene, the embodiment of the application further provides the robot device, and the robot device may perform live-action shooting on the commodity display situation in the physical space location to generate the virtual space scene, and may perform daily update, and the like. In addition, the robot equipment can also provide shopping guide service for users entering the physical space places, and comprises the steps of carrying out subject identification in the process of collecting the image stream of the physical commodities displayed in the shopping mall, generating an information card after obtaining the commodity search result, and then mapping the information card to the image stream in an AR (augmented reality) mode for displaying. In this way, a closed loop of "shopping mall on line" and "shopping mall under line" can be formed by the robot device, and the service of searching for goods based on images can be provided for the user whether on line or under line.

Moreover, no matter the user terminal device is used for "shopping mall going on line", or the robot device is used for "shopping mall going off line", the image frame is dynamically changed, so that the position, the state and the like of the main image in the image frame are changed accordingly. Therefore, in the embodiment of the application, different types of information cards can be provided for different body states, and the information cards can be made to move along with the body.

In the method, for the problem that the motion track encountered during the process that the information card follows the motion of the main body is discontinuous, in the preferred embodiment of the application, the problem can be solved by means of segment fitting. The collision problem between the information card and the edge of the screen can be solved by presetting two relative position relations between the information card and the center point of the main body and switching the relative position relations when the collision exists. Aiming at the collision problem between the information cards, the priority between different types of information cards can be determined through the priority relation between different main body states, and then the information card with lower priority can be determined as the information card needing to be moved. Or a plurality of layers can be created, and information cards corresponding to different main body states can be displayed on different layers.

Moreover, to in XR glasses scene, can also be through combining user "stretch out one's hand to go to grab" information card in order to accomplish the interactive mode of commodity "purchase additional", can further promote interactive interesting and user's participation.

Of course, it is not necessary for any product to achieve all of the above-described advantages at the same time for the practice of the present application.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required in the embodiments will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of a system architecture provided by an embodiment of the present application;

FIG. 2 is a flow chart of a first method provided by an embodiment of the present application;

FIG. 3 is a schematic diagram of a display interface provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a timeline provided by an embodiment of the present application;

FIG. 5 is a flow chart of a second method provided by embodiments of the present application;

FIG. 6 is a schematic diagram of a first apparatus provided by an embodiment of the present application;

FIG. 7 is a schematic diagram of a second apparatus provided by an embodiment of the present application;

fig. 8 is a schematic diagram of an electronic device provided in an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some embodiments of the present application, and not all embodiments. All other embodiments that can be derived from the embodiments given herein by a person of ordinary skill in the art are intended to be within the scope of the present disclosure.

In the embodiment of the present application, a new product search mode may be provided for a user, in which a product search may also be performed based on an image, but a virtual space scene may be established by a system in advance according to a product display condition in an entity space location (e.g., a mall, etc.) without uploading or taking a photo/video by the user, and the virtual space scene may be created by performing live-action shooting on the entity product display condition in the entity space location, performing three-dimensional reconstruction on a space structure of the entity space location, and then splicing and matching live-action shot images into the three-dimensional reconstructed virtual space structure. That is, in the virtual space scene created in the embodiment of the present application, it is not necessary to create a virtual 3D model for the commodity in the scene, but the virtual space scene is expressed by an image photographed in a real scene. In this way, the virtual space scene can be made more realistic, and the maintenance cost is also lower.

In addition, in the case where the virtual space scene is created, the embodiment of the present application may provide a function of searching for product information based on a live-action image in such a virtual space scene. Specifically, in the process that the user browses the Virtual space scene in a VR (Virtual Reality) manner, an image at a certain viewing angle in the Virtual space scene is presented in front of the user, and the image is composed of a live-action shot image. Of course, the user can switch the viewing angle by sliding the screen or the like to view the images at other viewing angles in the virtual space scene. In the above process, subject recognition may also be performed from the image frame therein (where the subject in the image recognition generally refers to a photographic subject located in the foreground of the image, and in the embodiment of the present application, a specific subject is generally an image of a physical commodity), and a related commodity information search result is obtained based on the recognized subject. Then, an information card to be displayed can be generated according to the commodity information search result, and the information card is mapped to the image frame for displaying. For example, the price of the goods, the preferential information and the like searched specifically can be displayed in the specific information card.

That is to say, in the embodiment of the present application, the user can obtain the experience of "shopping mall online" through the virtual space scene, and can view not only the display of the merchandise in the physical space such as the specific shopping mall, but also more additional information about the merchandise in the information card displayed by the merchandise. In addition, the live-action shot image carried in the form of the virtual space place also serves as an entrance for searching the commodities through the image, and a user can visit a shopping mall on line and acquire a commodity searching result in a mode of searching the commodities in a picture, so that a brand-new initiating mode and experience about searching the commodities in the picture are provided for the user. For merchants, the passenger flow can be changed from 'one city' to 'whole internet men', which is beneficial to greatly increasing the sales volume.

The physical space location may be a mall, and when a corresponding virtual space scene is created, because the display condition of the physical goods needs to be photographed in the physical space location, in a preferred embodiment of the present application, a robot device may be further provided, and the robot device may "walk" in the mall at a non-business time of the spatial location such as the mall, enter each store in the mall to photograph the display condition of the goods therein, and then the server may generate the specific virtual space scene according to the photographed image uploaded by the robot.

In addition, the robot device can also provide shopping guide service for consumer users during business hours of a shopping mall, in the process of providing the shopping guide service, real-time image streams of physical commodities displayed in the shopping mall can be collected, subject identification can be carried out based on the real-time image streams, commodity search is carried out based on the identified subjects, then, information cards can be generated in the same way, and the information cards are displayed in the modes of AR or holographic projection and the like.

Therefore, a closed loop of an online shopping mall and an offline shopping mall can be achieved through the robot device, namely, in the process that a user visits the shopping mall online, the robot device can search more additional information about commodities actually seen by the user for the user, generate an information card and display the information card to the user in an AR (augmented reality) or holographic projection mode and the like. In addition, a virtual space scene can be created through a live-action image acquired by the robot equipment, and the user can have an experience of "online shopping mall" in the form of VR. In addition, in the process of 'shopping mall line shopping' by the user, more additional information about commodities seen in the virtual space scene can be searched for the user through the terminal equipment of the user, an information card is generated, and the information card is matched with the VR picture for displaying.

In addition, no matter the information card is displayed in an AR mode through the robot device or the information card is displayed in a VR display process through the user terminal device, since an image stream collected by the robot device dynamically changes along with the movement of the robot device, the virtual space scene displayed in the terminal device can also change the viewing angle, move forward and backward and the like through sliding a screen, rotating the terminal device and the like, and therefore, the image of the virtual space scene displayed by the terminal device also dynamically changes. Accordingly, with respect to the subject identified therefrom, the position and state thereof in the image frame (including the size of the subject image, etc.) are also changed in real time. Therefore, in the embodiment of the application, the position, the state and the like of the main body can be recorded in real time, a plurality of different types of information cards can be defined in advance, and the different types of information cards can be displayed in different main body states. The subject state may be specifically determined according to the position of the subject in the image, an area ratio between the subject image and the screen, and the like, where the specific area ratio is changed according to the distance of the subject from the lens. For example, the body state can be divided into a discovery state, a presentation state, an interaction state, wherein the presentation state can be subdivided into a first presentation state, a second presentation state, and so on.

In addition, in the case where the information card is mapped into the image frame as described above, and a plurality of different types of information cards corresponding to different subject states are involved, it is necessary to handle presentation and the like of the information card on the client side (robot apparatus side, or user terminal apparatus side), and some problems may be encountered in this respect. For example, if it is necessary to make the information card move along the main body, how to make the movement track of the information card smoother and smoother under the condition that the end-side performance can be supported. Or, since the information card needs to occupy a certain area, but may collide with the edge of the screen during the process of following the movement of the main body, in the case that a plurality of main bodies are recognized in the image frame, a plurality of information cards need to be mapped into the image frame, and at this time, a plurality of information cards may also collide with each other. When such a collision occurs, the content in the information card may be blocked by the edge of the screen or other information cards, and how to solve the problem needs to be considered. In addition, since it is necessary to switch different types of information cards when the main body state changes, which is usually caused by a change in the distance between the subject and the lens, or by changing the angle of view, if the distance or angle of view is exactly at the boundary between the two main body states, a shaking phenomenon may occur. That is, the information card may repeatedly switch back and forth between the two types quickly, and so on. The application provides corresponding solutions for the problems in real time.

In addition, in specific implementation, besides the above control on the movement and switching processes of the information card, the prepositioning of some "action points" can be realized based on the information card, for example, the specific "action points" can include adding commodities into a "shopping cart". That is to say, the user can directly implement the additional purchase of the goods through the information card without entering a specific goods search result list page or goods detail page for additional purchase. Specifically, if the method is implemented in a mobile terminal device such as a mobile phone, an operation option specifically for adding a displayed commodity into a "shopping cart" may be provided in the information card, and a user may complete a shopping adding operation by clicking the operation option.

Or, in the case of "shopping online" through VR or MR (mixed reality) glasses, the additional purchase of the merchandise displayed on the information card can also be realized through a richer interactive form. For example, the user may grasp the information card and move the information card in a certain target direction (e.g., move the information card to the chest of the user, etc.) by using a handle of the MR glasses or by making a certain gesture, etc., so as to complete the purchase of the goods displayed on the information card, etc.

From the perspective of system architecture, referring to fig. 1, the embodiment of the present application may relate to a client and a server in a commodity information service system, where the client in the embodiment of the present application may include an App (application program) installed on a side of a terminal device such as a mobile phone, or may further include an App installed in a VR/MR glasses device, and the like. In addition, since live-action shooting during the virtual space scene creation process is also involved, in an alternative embodiment, a robot device may be provided, through which live-action shooting is performed on the merchandise display situation in the physical space location (which may include a plurality of subspaces, for example, a plurality of specific physical stores). The server side can create a virtual space scene according to the live-action shooting image, the client side can display the virtual space scene in a VR mode, subject identification is carried out from the live-action shooting image frame, a commodity search result is obtained from the server side, and an information card is generated to be mapped back to the virtual space scene for display. In addition, the robot device can provide shopping guide service for users entering the physical space place, and provide more additional information about commodities in the shopping mall in an AR manner in the shopping guide process. In addition, regarding the display control of the information card in the moving or switching process, including image stream processing, card rendering processing, body following processing, card anti-collision processing, card switching processing, and the like, may also be completed by the client side. The server is mainly used for creating a virtual space scene and providing specific commodity search results, and the client can generate specific information cards according to the specific commodity search results.

The following describes specific implementations provided in embodiments of the present application in detail.

Example one

First, from the perspective of a client on a user terminal device side in this embodiment, a method for providing information of a product search result is provided, and referring to fig. 2, the method may include:

s201: responding to a request for browsing commodity information in a Virtual Reality (VR) mode, and displaying a pre-generated virtual space scene based on VR technology, wherein the virtual space scene is generated in the following way: and performing live-action shooting on the display condition of the entity commodity in the entity space place, performing three-dimensional reconstruction on the space structure of the entity space place, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating the virtual space scene.

The physical space may be a shopping mall, or an independent shop. The embodiment of the application can carry out live-action shooting on the commodity display condition in the entity space place, carry out three-dimensional reconstruction through the space structure of the entity space place, splice and match live-action shot images into a three-dimensional reconstructed virtual space structure, and generate the virtual space scene. That is to say, in the embodiment of the present application, the virtual space scene is generated by stitching live-action images, and 3D modeling and the like are not required for a physical commodity in a physical space location, so that the cost is relatively low, and meanwhile, the method is more realistic.

After the virtual space scene is created, the virtual space scene may be published online, for example, if an online store is associated with a physical space location, an entry for browsing in a VR method may be provided in a page such as a store home page, and accordingly, a user may initiate a request for browsing commodity information in a VR method through the entry. Or, in another mode, a function module such as "shopping mall online" may also be provided in the client, and an entry of the function module is provided through a page such as a home page of the client, and in a specific receiving page, access entries of multiple virtual space scenes may be collectively displayed, so that a user selects a specific virtual space scene during accessing the receiving page, and initiates a request for browsing commodity information in the virtual space scene in a VR mode, and so on.

Regarding the creation of the virtual space scene, in one mode, the commodity display situation in the physical space place can be shot in a real-time mode through a manual shooting mode, and uploaded to the server, and the server generates the corresponding virtual space scene. Alternatively, it is contemplated that the display of items in a physical space venue may change frequently, for example, a store may change the items displayed in the window on a daily basis, and so forth. At this time, it may be necessary to frequently perform live-action shooting again and update the virtual space scene. Therefore, in another mode, the embodiment of the present application further provides a robot device, which may perform live-action shooting on the display situation of the physical commodity in the physical space location through the robot device during the non-business hours of the physical space location, so as to obtain a specific live-action shot image.

In this case, if the physical space is a mall or the like, in which a plurality of different subspaces are usually distributed, for example, the physical stores, the physical goods displayed in the physical space include the physical goods displayed in the specific physical stores. Therefore, when the live-action is performed, the actual-action may be performed by the robot device on the actual commodity display conditions in a plurality of different actual stores in the actual space location, in particular, during the non-business hours in the actual space location. That is, the robot device may be controlled, or in the case that the robot has machine vision, the robot may "walk" in a shopping mall, enter each physical store to take live-action pictures of the merchandise display situation therein, and upload the pictures to the server. In which, some exhibitions and the like may be arranged in advance in a specific physical shop to support the live-action shooting. Wherein, also can go on according to appointed standard to the commodity display mode on the stand, for example, can separate certain distance between the different commodity (so that follow-up VR show in-process, reduce the collision between the different information cards), to the commodity of clothing class, need can shoot the front (so that discern the main part, and acquire accurate search result), etc..

In the case that the same physical space location includes a plurality of physical shops, particularly when three-dimensional reconstruction of a spatial structure is performed, three-dimensional reconstruction of spatial distribution structures of the plurality of physical shops in the physical space location may be performed, so as to better restore a real scene of a mall.

However, how to perform the live-action shooting specifically and how to construct the virtual space scene based on the live-action shot image do not belong to the important point of the embodiments of the present application, and the details are not described here.

S202: and carrying out subject identification from the live-action shooting image frame corresponding to the current display visual angle in the virtual space scene.

In the step S201, the virtual space scene is displayed by VR technology, and in the process of displaying the virtual space scene, subject identification may be performed on the image frames therein. The real-scene shooting image frame refers to: the virtual space scene is usually a 3D scene, and the user terminal screen is usually rectangular, so that only part of images in the virtual space can be viewed at the same time, and the user can switch the viewing angle by sliding the screen or rotating the terminal device, or can click a forward/backward button in the scene, etc., to advance or retreat in the virtual space scene. In the process, the visual effect is similar to the process of dynamically acquiring the image stream through the camera, and a user can browse multiple frames of pictures, wherein each frame can be called an image frame. In addition, in the embodiment of the present application, the images in the virtual space scene are formed by splicing the live-action images, so that each frame of image can be referred to as a live-action image frame.

The subject recognition may be mainly to recognize a subject belonging to the foreground from the image frames, for example, if a certain image frame is obtained by photographing a commodity displayed in a certain area in a certain shop, the commodity may be recognized as a subject, and a background portion in the corresponding shop may be captured in the image frame but may not be recognized as a subject. Of course, in the same image frame, a plurality of subjects may be recognized, for example, in the case where a plurality of articles enter the shooting range at the same time, a plurality of subjects may be recognized, and the like. The specific subject identification result may include the number, coordinates, length, width, category, whether a search request may be initiated to the server, and the like of the subject. The client may record such recognition results for subsequent use in the generation, presentation, and motion/switching control of the information card.

The number of the main body is mainly used for marking the specifically identified main body, the mark is given by an algorithm, and in the identification result of each frame, the number corresponding to the same main body is the same. In this way, when a plurality of subjects are identified in an image frame, each subject can be distinguished by a subject number, and subsequent information card tracking and the like can be realized.

The coordinates of the subject mainly refer to the coordinates of the center point of the subject in the screen, and the length and the width refer to the maximum width and the maximum length of the subject image in the screen. In the process of dynamically changing the view angle, the coordinates, length and width of the same subject are constantly changing, and can be identified by the algorithm deployed on the client side. Accordingly, recording can be performed by the client.

The subject category mainly refers to a commodity category to which the photographic subject corresponding to the subject image may belong, that is, to identify what item the photographic subject is specifically, for example, a cup, an earphone, and the like. The information can be used for generating an information card of the "pocket bottom", that is, when a specific commodity search result cannot be obtained, information such as a name of a commodity can be displayed in the information card according to the commodity category to which the current main body belongs, and the like.

Whether to initiate a search request to the server side means that specific commodity search results can be obtained from the server side, however, since the size, definition and the like of the subject image are constantly changed in the process of dynamically changing the view angle in the virtual space scene or executing forward and backward operations, when the subject image is still small, the server side may have difficulty in giving accurate commodity search results based on the subject image, and therefore, the search request may not be submitted to the server side. Then, if the area of the main image is large enough and clear enough, it can be determined that a commodity search request can be initiated to the server. In addition, as the subject image becomes clearer, the server may be able to give more accurate search results, and therefore, a product search request may be re-initiated to the server, and so on. And the above-mentioned various decisions can be completed by the aforementioned client-side algorithm in the process of subject recognition.

After the algorithm identifies the information, the information can be recorded at the client side, and the position information of the main body can be determined according to the information such as the coordinates of the center point of the main body.

In practical application, when a virtual space scene is displayed by using a VR technology, a user may change an angle of view by sliding a screen or rotating a terminal device, or perform operations such as forward and backward movements, and an image displayed on a screen of the terminal device may also dynamically change along with the change of the angle of view (similar to a presentation effect of real-time image stream acquisition by using a camera). Therefore, the process of performing the subject recognition from the image frames is also dynamically continued, and when the recognition is performed at a specific time, the subject recognition is performed from the image frame displayed in front of the user at the current time.

In addition, in a specific implementation, during the process of dynamically changing the view angle or performing forward and backward operations on the virtual space scene, the subject can be identified from a plurality of image frames, and the position and state change of the subject in the plurality of image frames can be acquired. Therefore, the information card to be displayed can be generated subsequently according to the commodity information search result and the state of the main body, and different main body states can correspond to different types of the information card.

The state of the subject, that is, the position of the subject in the image frame and/or the area ratio of the subject image in the screen, is referred to herein. For example, in one mode, the subject state may be determined simply based on the area ratio of the subject image in the screen. Specifically, for example, when the subject state is divided into the finding state, the first display state, the second display state, and the interaction state, the area ratio (assumed to be represented by the letter M) occupied by the subject image in the screen is small, for example, a < M < b. At this time, it may be possible to identify only what kind of item the body is, and it is impossible to determine which actual vendable items are related to in the system, and therefore it may be determined that the body is in the "discovered state". Then, as the area occupied by the main body image in the screen is gradually increased, for example, b is less than or equal to M < c, the image is gradually clear, when the degree of commodity searching based on the main body image is reached, the searching can be initiated to the server, and correspondingly, the main body enters the first display state. Then, when the area of the main body image is further increased, for example, c is less than or equal to M and less than d, and the image is clearer, a search request may be initiated to the server again (or, a request may not be required again, and may be determined according to actual needs), and at this time, the main body may enter a second display state. Thereafter, if the subject image area is increased still further, e.g., d ≦ M < e, for a certain length of time (e.g., 3S), the subject enters the interactive state, and so on.

Or, in another case, in determining the subject state, in addition to the factor of the area ratio occupied by the subject image, the coordinate factor of the subject image may be considered. For example, with respect to a plurality of predefined subject states, in addition to the area ratio of the subject image corresponding to each subject state, position information corresponding to each subject state may be defined. For example, a "discovery frame", "presentation frame", "interaction frame", etc. may be predefined for the discovery state, the first presentation state, the second presentation state, and the interaction state. In this way, a subject can be determined to be in a finding state only when the area ratio a < M < b of the subject image and the subject center point is located inside the "finding frame" and outside the "presentation frame". Similarly, the subject can be determined to be in the display state only when the area ratio b of the subject image is less than or equal to M < c or c is less than or equal to M < d, and the subject center point is located inside the display frame and outside the interactive frame. If the area ratio d of the main body image is less than or equal to M and less than e, and the center point of the main body is located in the interactive frame body, the main body can be determined to be in an interactive state, and the like.

S203: a search result of the commodity information related to the identified subject is acquired.

After a subject is identified from the image frame, a product information search result related to the identified subject may be acquired. Specifically, as described above, the relevant search result may be obtained from the server, or, if the terminal device has a local cache, the terminal device may also read the relevant search result from the cache, and so on. When the server side searches, the main body image can be specifically intercepted from the image frame, a search request is submitted to the server side, and the server side retrieves a search result which specifically meets the conditions.

In addition, since the client-side algorithm can also identify the number, the category, and the like of the subject, when a search request is submitted, the identification result about the number, the category, and the like of the subject can also be submitted to the server, and the server can search out the product search result related to the subject image according to the information. When the product search result is returned, information such as a specific body number may be added.

In the embodiment of the present invention, the information of the product displayed in the physical space location may be previously distributed to the server by the relevant merchant and stored by the server, so that when a product search is performed based on the subject image identified from the live-action image, the product information corresponding to the physical product displayed in the physical space location may be directly searched. Of course, in the specific implementation, the commodity database of the server may also have commodities of the same or similar money as the entity commodity, so that the commodity information may also be provided in the search result for comparison by the user.

S204: and generating an information card to be displayed according to the commodity information search result, and mapping the information card to the image frame for displaying.

After the commodity information search result is obtained, an information card to be displayed can be generated according to the commodity information search result. The specific information card is mainly used for displaying some text information, for example, including information such as a commodity name, a price, user evaluation and the like acquired from a commodity search result. And then can be mapped into a virtual space scene for showing. In a preferred mode, the position of the corresponding main body in the virtual space scene can be mapped to be displayed. Therefore, the user can browse the commodities in the virtual space scene in a VR mode and can acquire the additional information of the specific commodities from the information card. It should be noted that, since the virtual space scene in the embodiment of the present application is generated based on a way of stitching live-action images, although the method has advantages of low maintenance cost, high degree of reality, and the like compared to a way of reconstructing a 3D model of a commodity, there is a deficiency in the richness of information provided in the virtual space scene. In the embodiment of the application, the subject recognition can be performed based on the live-action shot image in the virtual space scene, and after the commodity search result is obtained, the commodity search result is mapped to the virtual space scene in the form of the information card, so that the information richness is improved. That is, the user can obtain a spatial scene with higher fidelity and simultaneously obtain richer information.

As described above, in the process of displaying the virtual space scene, the user may change the viewing angle, so that the image content displayed on the screen dynamically changes, and accordingly, the identified position and state of the subject may also change. Thus, in a preferred embodiment of the present application, different types of information cards may also be provided based on different subject states.

The status of the subject may include a discovery status, a presentation status, an interaction status, and the like, as described above. The type of the information card can be divided mainly from the aspects of the size of the card, the content richness and the like. The corresponding information card type may be defined in advance in various different body states. For example, when the main body just starts to enter the display state, the content in the information card may be from the commodity search result information, but the information card may be small, and the amount of information displayed may also be small. Then, as the subject image further increases, it is possible to switch to a larger-sized information card in which the information to be presented can be richer.

In a word, the information cards can be of various types, the information cards can be changed into small ones and larger ones along with the increase of the area ratio occupied by the main body image, and the displayed information can be richer. For example, as shown in fig. 3 (B) (for convenience of illustration, only one subject image is shown in the figure), assuming that the currently recognized subject is a headphone, when the percentage of the subject image reaches 40% (or other values), an information card of a smaller size may be displayed, in which information card of a smaller size, a name of a commodity, a reference price, representative user evaluation information, and the like may be included only. As shown in fig. 3 (C), after the subject image becomes large, for example, when the image occupation ratio reaches 60% (or other numerical values), a larger-sized information card may be displayed, in which richer information about performance introduction and the like of the merchandise may be included in addition to the information displayed in the aforementioned smaller-sized card. In addition, when information of a certain commodity in the commodity search result is displayed in a specific information card, besides the information related to the commodity introduction, some marketing information may be included, for example, a certain merchant configures a certain coupon for a certain commodity, and the marketing information may be passed through the information card to the user, so that the user may know the special benefit information about the specific commodity before entering a specific search result list page, and thus, the click rate, the conversion rate, and other indexes of the information may be improved. Or, since the object in the specific live-action shot image is a real article and may be an article that the user has purchased, the merchant may also configure a user right such as "another bottle" for some special articles (e.g., beverages, etc.), at this time, the user right information may be provided by an information card, and the user may obtain the right by clicking the information card, so as to promote interactivity, and so on.

Of course, as mentioned above, in a specific implementation, the identified subject may also include a discovery status, that is, the image of the subject has a small area ratio, which is not enough to accurately provide an accurate product search result, but can roughly identify the category of the object. Therefore, a corresponding information card can also be generated based on the category information. For example, as shown in fig. 3 (a), recognition is started when the proportion of the subject image reaches 20%, but only the object category information may be displayed in the information card displayed at this time, and so on. Of course, in practical applications, when the subject is in the discovered state, the object type recognized initially may be wrong, and subsequently, the type recognition result may be corrected as the subject image gradually increases, and the content displayed in the information card may be updated.

In addition, the recognized main body can also comprise an interaction state, at the moment, the area ratio of the main body image is large, the image is clear enough, and an information card of the interaction state can be displayed. Alternatively, the information presentation form may also be changed, for example, the presentation may no longer be in the form of an information card, but rather a 3D model of the item may be mapped into the real-time information stream. There are many specific interaction forms, for example, in one mode, after entering an interaction state and keeping for a certain time (for example, 3S), the user may jump to a search result list page for display, or even jump to a detail page of a specific product currently being displayed for display, and so on.

It should be noted that, in the process of displaying the additional information of the specific product through the information card, some "action points" may also be provided through the information card, for example, the action points may include adding the product to a batch settlement product set (also known as a "shopping cart") and the like. Specifically, if the method is implemented in a mobile terminal device such as a mobile phone, an operation option for adding a commodity corresponding to a specific information card into a "shopping cart" may be provided in the information card. Therefore, if the user is interested in the commodity corresponding to the information card, the user can directly click the operation option to complete the additional purchase of the commodity without entering a search result list page firstly and then entering a specific commodity detail page for additional purchase, and therefore the operation path of the user can be shortened.

Or, the scheme provided by the embodiment of the application can be used in mobile terminals such as mobile phones and also can be used in VR or MR glasses equipment. That is, the functions provided by the embodiments of the present application can be implemented in the relevant application program on the VR or MR glasses device side. In this way, in the process of "shopping the mall online" when the user wears the VR or MR glasses device, in this scenario, the VR or MR glasses have an advantage over a mobile terminal such as a mobile phone in that the VR or MR glasses do not need to be held by hands, and the two hands of the user can be released, and meanwhile, the hands can perform some control operations on the screen content through the handles associated with the VR or MR glasses, or perform control operations on the screen content by making some gestures, and so on. Therefore, in the scenario of such VR or MR glasses, regarding the operation of adding an article to the "shopping cart", it may not be necessary to provide the user with an associated additional purchase button in the information card, but it may be implemented by detecting the operation of grasping the information card and moving toward a target direction (e.g., the chest direction of the user, etc.) performed by the user. That is to say, in the process of watching a commodity and an information card in a live-action image in a virtual space scene through VR or MR glasses, if a user is interested in the commodity displayed on the information card, the user can perform an operation of grabbing the information card and moving the information card in the chest direction through a handle or a gesture, so that the operation of "buying in advance" on the commodity can be triggered. Therefore, the interaction interest and the participation degree of the user can be further improved by combining the interaction mode that the user stretches out and grasps the information card.

In addition, in the case where the robot device is deployed in the physical space location, in addition to live-action shooting of the product display situation in the physical space location by the robot device, the robot device may also provide a shopping guide service for a user who enters the physical space location for shopping. Specifically, the shopping guide service based on the AR mode may be provided to the user entering the physical space location through the robot device during the business hours of the physical space location. The specific shopping guide service may include: and acquiring a real-time image stream of the display condition of the entity commodity in the entity space place, identifying a subject from the real-time image stream, acquiring a commodity information search result related to the identified subject, generating an information card to be displayed according to the commodity information search result, and displaying the information card.

That is, after entering the physical space, the user can visit the shopping mall in the "leading" area of the robot device, and the robot device not only can help the user quickly locate the specific shopping mall to be visited, but also can search out more additional information about the goods displayed in the shopping mall on line for the user in an AR manner, so as to help the user make a shopping decision. For example, when a user visits a certain shop, the AR function of the robot device may be started, and at this time, the robot device may start the camera to perform real-time image stream acquisition, and may perform subject recognition during the process of acquiring the image stream.

The process of the subject recognition is similar to the process of the subject recognition from the live-action shot image of the VR scene, and the position, the state and the like of the subject can be dynamically determined, and different types of information cards can be generated for display. It should be noted here that there are many ways for the robot device to display such information cards, for example, in one way, the information cards can be mapped to the positions of the subjects in the corresponding image frames for display by means of AR. That is, the user can view the AR picture through the display screen of the robot device, and the AR picture includes the image collected in real time and the information card.

Or, in another mode, the client of the robot device may also display the information card in a holographic projection mode. That is, the user can view the specific information card in a space other than the display screen of the robot device. In addition, in this way, the 3D model of the commodity corresponding to the information card can also be displayed in a holographic projection way. That is, the user can obtain the experience that the commodity is "standing up" in front of the user, the 3D model can also automatically rotate and the like, so that the user can see the full view of the commodity from more visual angles, and simultaneously can obtain more additional information about the specific commodity through an information card displayed nearby, wherein the additional information comprises the commodity name, the price, user evaluation information and the like.

In the process of displaying the information card by the robot device in the AR or holographic projection manner, the operation option for adding the specific commodity corresponding to the information card to the batch settlement commodity set (that is, what is called a shopping cart or the like) may be used. In this way, the user can complete the "buy-in" operation on the goods by clicking the operation option, and the like.

In addition, the robot device may provide a shopping guide service for the user, and may also provide information such as reference opinions for the user, for example, the user may ask the robot device "good and not good" in the process of trying or trying on a certain commodity, the robot device may judge the trial or trying-on effect of the user through machine vision, related algorithms, and the like, and may also give a comparison opinion on similar commodities that the user tries or tries to pass through in the current shopping process, and the like.

The implementation scheme that the image main body is identified through the virtual space scene or the robot device in the process of shopping at the upper line and the lower line and the information card is provided according to the commodity searching result based on the main body image is introduced. In one mode, after a specific information card is generated, the specific information card may be mapped to a position where a main body is located in an image frame (an image frame currently displayed in a screen in a virtual space scene, or a current image frame in an image stream acquired by a robot device) for displaying. However, in the actual implementation, since the position of the main body is constantly changing, it is necessary to move the information card along with the main body. It is also possible for the information card to collide with the edge of the screen, or other information cards, during its movement with the body. In addition, when the state of the main body changes, switching of the type of the information card is involved, and the like. In the embodiment of the present application, solutions are respectively provided for the above.

In the process of moving the information card along with the main body, because the algorithm can perform main body identification on each frame of image in the image stream, the position of the main body in each frame of image can be changed, and the identified main body position is a plurality of discrete points. At this time, if the position of the information card is updated directly according to the body position identified in each frame, the motion trajectory of the information card may not be smooth enough, and in addition, it is a great challenge for the client performance.

Therefore, in the embodiment of the present application, a piecewise fitting implementation is adopted. Specifically, the client may store the algorithm identification result, and may read the latest position of the identified main body at preset time intervals in the process that the information card moves following the change of the position of the main body. Wherein the time interval may be related to the human eye persistence time. For example, in a specific implementation, the time interval may be 0.35S, that is, the latest position of the main body is read every 0.35S, and after the latest position is read, an animation that smoothly moves from the current position to the latest position may be generated for the information card currently being displayed. The position of the information card may then be held constant at this latest position until a new latest position is read in the next cycle, generating a new animation.

For example, as shown in the time axis of fig. 4, assuming that a certain information card is displayed from time a, the information card may be displayed at the position according to the position of the body at time a, and animation may not be performed for a while, and then animation update may be performed once at intervals of n1 s. For example, the time interval from time a to time B is n1s, and at this time, although the algorithm updates the subject position for a plurality of times, the information recorded on the client side is also updated continuously, but the animation is not updated, that is, the information card is still at the position of time a. When the time B comes, the information card starts to move by using a main body position which is updated recently, and the specific moving mode can be that a translation animation with the duration of n1s is made. That is, the information card smoothly moves from the main body position at time a to the position at time B along a straight line. And then, the information card keeps unchanged at the position of the time B until the time C arrives after n1s, and then the information card moves from the position B to the position C through the n1 s-long translation animation, and so on. That is to say, in this way, the motion trajectory of the information card is a linear motion divided into a plurality of segments, and the linear motions of the segments are connected together to realize the fitting of the motion trajectory of the main body, so that the motion of the information card can be smoother and smoother, and the pause phenomenon can be reduced. And the translation animation of multiple segments is more friendly to the performance of the client.

The problem about the collision of the information card and the edge of the screen, that is, because the information card can be generally rectangular, in the process of following the movement of the main image, the collision of the information card and the edge of the screen can occur, so that part of the content of the information card is shielded and can not be displayed.

In order to solve this problem, in the embodiment of the present application, the relative position relationship between the information card and the main body anchor point (specifically, the position of the main body center point may be determined, for example, the main body center point may be directly used as the anchor point, or the anchor point and the main body center point may be on the same horizontal or vertical straight line, etc.) may be defined in advance as multiple types, for example, it is assumed that two different types of relative position relationship may be defined, namely, the first relative position relationship and the second relative position relationship. Specifically, the upper right and lower left can be mentioned. That is, when the information card is displayed, the information card may only appear in the lower right or upper left position of the anchor point. Therefore, in the process that the information card moves along with the position change of the main body, whether the information card collides with the edge of the screen in the current relative position relation state can be judged, and if so, the information card is switched to other relative position relations. For example, the information card is originally at the lower right position of the anchor point of the main body, but the information card may collide with the right edge of the screen along with the movement of the main body, so that part of the information in the information card may be blocked. At this time, the information card may be switched to the upper left position. When judging whether can collide with the screen edge, if the current information card is displayed at the right lower side of the main body anchor point, whether the information card can collide with the right edge of the screen is mainly judged, specifically, the X-axis direction coordinate of the center point of the main body can be added with the length of the card, and whether the obtained result can exceed the right edge of the screen is judged, and the like.

Of course, in specific implementation, if it is found that the information card exceeds the right edge of the screen when in the current relative position relationship, the information card may also exceed the left edge of the screen after being switched to other relative position relationships of the main anchor point.

Under the implementation mode of performing piecewise smooth fitting on the motion track of the information card, the latest position of the main body can be read every n1s, and after the information card is moved, whether the information card exceeds the edge of the screen or not is judged.

In addition, as shown at 31 in fig. 3 (B), in the process of specifically presenting the information card, a specific anchor element may also be presented on the subject image. In the process of switching the relative position relationship, the anchor point element may be hidden first, and after the switching of the relative position relationship is completed, the anchor point element may be switched to the visible state. This may make the process of switching more aesthetic.

In the practical application, the situation that a plurality of main bodies are identified in the same image frame may also occur, at this time, if the corresponding commodity information search results are respectively obtained and the corresponding types of information cards are respectively generated according to the respective states, the plurality of information cards may also collide with each other in the process of mapping the plurality of information cards into the image frame in real time and moving along with the position change of the respective corresponding main bodies, that is, the situation that one information card blocks the content of another information card may occur.

For such a situation, in the embodiment of the present application, it may also be determined whether a plurality of information cards collide with each other, and if so, the position of one or more of the information cards may be moved to eliminate such a blocking phenomenon, and as to which information card is specifically moved, in the embodiment of the present application, the information card display position corresponding to the lower-priority body may be moved according to the difference in the body state priority corresponding to the information card that collides with. That is to say, since a plurality of different subject states are defined in the embodiment of the present application, and the subject states are influenced by the size of the subject image, etc., the states of a plurality of subjects in the same image stream may be different, and the larger the subject image area ratio is, the more likely the subject image is to be an object that the user wants to search for most, and therefore, the corresponding information card may be displayed with higher priority. Therefore, various different main body states can be sorted according to the priority levels, and when two information cards collide, the main body corresponding to the information card with the lower priority level can be removed.

It should be noted that, if a certain information card is moved, the information card collides with the edge of the screen, and the edge of the screen blocks the information card, and at this time, the information card may not be moved. Therefore, in order to better reduce the influence caused by mutual collision between different information cards (mainly shielding the contents in part of the information cards), a plurality of different layers can be created, each layer is used for displaying one of the information cards corresponding to the main body state, and the upper and lower layer relationships of the layers can be sorted according to different priorities of the main body states. For example, a first layer from top to bottom may be used to carry contents such as information cards corresponding to the interaction state, a second layer may be used to carry a "big card" of the display state, a third layer is used to carry a "small card" of the display state, a fourth layer is used to carry information cards of the discovery state, and so on. Thus, when a certain main body is in a certain state, a specific information card can be displayed in the layer corresponding to the state. When different types of information cards are switched and displayed for the same main body, the layers can be switched simultaneously. For example, a certain main body is in a first display state before, and the type of the information card is a "small card" in the display state, at this time, the information card may be displayed in the third layer; at some point the body state changes to the second presentation state, the information card type switches to "big card" in the presentation state, at which point the "big card" may be mapped to the second layer in the image stream, and so on.

By the method, the information cards corresponding to different main body states are displayed in different layers, and the higher the priority of the main body state is, the higher the corresponding layer is. Therefore, if the information cards corresponding to two different main body states collide, the information card corresponding to the main body state with higher priority can be ensured to be positioned on the upper layer. Therefore, even if the information cards collide with each other and the information cards with lower priorities cannot be removed, the information cards corresponding to the main body state with higher priorities can be prevented from being shielded by the information cards corresponding to the main body state with lower priorities.

In addition, since a plurality of different subject states are defined in the embodiment of the application, different subject states can correspond to different types of information cards, and the subject state of the same subject in an image frame generally changes, the problem of switching the types of the information cards is also involved particularly in the display of the information cards. That is to say, in the process of mapping the information card to the position of the corresponding main body in the real-time image stream in real time for displaying, if the state of the main body changes, the type of the information card mapped to the real-time image stream is switched. For example, an information card with a smaller size is displayed for an entity at a certain time, and then, the area ratio of the image of the entity is increased, the image is clearer, and the information card with the larger size can be switched. For example, as shown in FIGS. 3 (B) and (C).

The switching of the information card type is triggered by a change event of the main body state, and the change of the main body state is related to the change of the area ratio of the main body image. For example, as described above, the area ratio is divided into four sections, each corresponding to one body state, and if a certain value is used as a boundary of various body states, at the boundary of two body types, a situation in which the information card "shakes" may occur. For example, it is assumed that the first display state corresponds to a smaller-sized information card (which may be referred to as a "small card" for convenience of description) when the subject image area ratio b ≦ M < c, and the second display state corresponds to a larger-sized information card (which may be referred to as a "large card" for convenience of description) when the subject image area ratio c ≦ M < d. If the area ratio of the subject image is changed around c, for example, if c =0.5, M may change back and forth between 0.49 and 0.51, and at this time, a phenomenon that "small card" and "large card" in the image frame are switched repeatedly and rapidly may occur, which may be referred to as a "jitter" phenomenon of the information card.

In order to solve the 'jitter' phenomenon in the information card type switching process, the embodiment of the application is realized by adding a buffer at the boundary of various main body states. For example, in a general case, the division manner of the various subject states may be:

a < M < b is the discovery state;

b < = M < c is to show a "small card" state;

c < = M < d is to show "big card" status;

d < = M < e is the interaction state.

However, in order to solve the "jitter" phenomenon in the information card type switching process, the buffer n may be added at the boundary, and in this case, the various body states may be divided in the following manner:

a < M < b is the discovery state;

b + n < = M < c is a display small card state;

c + n < = M < d is showing big card state;

d + n < = M < e is the interaction state.

In this way, it is possible to have a buffer of size "n" at the boundary between two different body states, instead of having a simple certain value as the boundary. In this way, if the user happens to have "hand shake" at the boundary between two states of a certain main body in the process of acquiring the image stream, as long as the variation of the image area ratio of the main body caused by the "hand shake" is within n, the switching of the information card type is not triggered, so that the problem of the "shake" of the information card at the boundary of the main body type is solved. Of course, in particular implementations, other determination parameters may be used to determine the subject state, including, for example, the size, resolution, etc. of the subject image.

In addition, in the process of specifically switching the information card types, the first type of information card is switched to the second type of information card, which is two different information cards, that is, the second type of information card is equivalent to replace the first type of information card, so that the switching process of the information card types can be controlled. For example, in a specific implementation manner, the switching process may be divided into two stages, where the first stage is configured to gradually decrease the transparency of the first type of information card currently projected in the real-time image stream (for example, the transparency may be gradually decreased from 1 to 0, that is, the first type of information card gradually disappears), the second stage is configured to project the second type of information card into the real-time image stream, and gradually increase the transparency of the second type of information card (for example, the transparency may be gradually increased from 0 to 1, that is, the second type of information card gradually appears), and so on.

In short, by the scheme provided by the first embodiment of the application, the physical commodity display condition in the physical space site can be shot in advance, the space structure of the physical space site is reconstructed in three dimensions, and the shot images are spliced and matched into the virtual space structure of the three-dimensional reconstruction to generate the virtual space scene. In this way, after a user initiates a request for browsing commodity information in a VR mode, a pre-generated virtual space scene can be displayed through VR technology, and subject recognition is performed from an image frame in the virtual space scene. Then, a commodity information search result related to the identified subject may be acquired, an information card to be displayed may be generated according to the commodity information search result, and the information card may be mapped to a position of the subject corresponding to the image frame for display. By the method, a user can obtain the experience of shopping mall online, the live-action shooting image frame contained in the virtual space scene can be used as the starting point of commodity searching with the picture, the commodity searching result is automatically obtained through main body identification and is mapped back to the virtual space scene in the form of the information card for displaying, so that the information in the virtual space scene created in the live-action shooting mode is richer, and meanwhile, the user can obtain brand-new commodity searching interaction experience.

In addition, in order to facilitate creation and update maintenance of the virtual space scene, the embodiment of the application further provides the robot device, and the robot device may perform live-action shooting on the commodity display situation in the physical space location to generate the virtual space scene, and may perform daily update, and the like. In addition, the robot equipment can also provide shopping guide service for users entering into the physical space places, including main body identification in the process of collecting the image stream of the physical commodities displayed in the shopping malls, and an information card is generated after the commodity search result is obtained, and then the information card can be mapped into the image stream in an AR (augmented reality) mode to be displayed. In this way, a closed loop of "shopping mall on line" and "shopping mall under line" can be formed by the robot device, and the service of searching for goods based on images can be provided for the user whether on line or under line.

Moreover, no matter the user terminal device is used for "shopping mall online" or the robot device is used for "shopping mall offline", the image frame is dynamically changed, so that the position, the state and the like of the main image in the image frame are changed accordingly. Therefore, in the embodiment of the application, different types of information cards can be provided for different body states, and the information cards can be made to move along with the body.

Moreover, aiming at the MR glasses scene, the interaction mode of commodity additional purchase can be completed by combining the information card which is held by the user, so that the interest of the interaction and the participation of the user can be further improved.

Example two

The second embodiment provides a robot apparatus, which can be applied to a physical space site;

the robot equipment is used for providing shopping guide service based on an AR mode for a user entering the physical space place; the shopping guide service comprises: and acquiring a real-time image stream of the display condition of the entity commodity in the entity space place, identifying a main body from the real-time image stream, acquiring a commodity information search result related to the identified main body, generating an information card to be displayed according to the commodity information search result, and displaying the information card.

In addition, the robot device may be further configured to perform live-action shooting on the display situation of the entity goods in the entity space location, so as to perform three-dimensional reconstruction on the space structure of the entity space location, and splice and match live-action shot images into a three-dimensional reconstructed virtual space structure to generate a virtual space scene;

EXAMPLE III

It should be noted that the solution provided in the embodiment of the present application may also be applied to other physical space places, for example, space places such as a vegetable garden, a zoo, a museum, and an exhibition hall may be photographed by taking real-image images of plants, animals, exhibited items, and the like in advance, and a corresponding virtual space scene may be generated based on the real-image images. In the process of accessing such a virtual space scene by a user, subject identification (specific subjects may be plants, animals, exhibited items, and the like) may be performed from live-action image frames in the virtual space scene, and a search is performed based on the identified subjects, and a search result about the specific subjects, for example, information that may be encyclopedic in particular, and the like, is obtained. And then, generating a specific information card based on the searched information, and mapping the specific information card to the position of the corresponding main body in the virtual space scene for displaying. In this way, when the user browses the virtual space scene, more additional information about the specifically seen main body can be acquired through the information card.

Therefore, the third embodiment further provides an information displaying method, and referring to fig. 5, the method may include:

s501: displaying a pre-generated virtual space scene based on VR technology, the virtual space scene being generated in the following manner: performing live-action shooting on an entity in an entity space site, performing three-dimensional reconstruction on a space structure of the entity space site, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene;

s502: performing subject identification from a live-action shooting image frame corresponding to a current display visual angle in the virtual space scene;

s503: acquiring an information search result related to the subject based on the identified subject image;

s504: and generating an information card to be displayed according to the information search result, and mapping the information card to the virtual space scene for displaying.

For the parts of the second and third embodiments that are not described in detail, reference may be made to the descriptions of the first embodiment and other parts of the present specification, which are not described herein again.

It should be noted that, in the embodiments of the present application, the user data may be used, and in practical applications, the user-specific personal data may be used in the scheme described herein within the scope permitted by the applicable law, under the condition of meeting the requirements of the applicable law and regulations in the country (for example, the user explicitly agrees, the user is informed, etc.).

Corresponding to the first embodiment, an embodiment of the present application further provides an apparatus for providing information of a search result of an article, and referring to fig. 6, the apparatus may include:

a VR display unit 601, configured to display a pre-generated virtual space scene based on VR technology in response to a request for browsing commodity information in a virtual reality VR mode, where the virtual space scene is generated in the following manner: the method comprises the steps of carrying out live-action shooting on the display condition of the entity commodity in an entity space place, carrying out three-dimensional reconstruction on the space structure of the entity space place, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene;

a subject recognition unit 602, configured to perform subject recognition from a live-action image frame corresponding to a current display perspective in the virtual space scene;

a product search result acquisition unit 603 for acquiring a product information search result related to the recognized subject;

and an information card display unit 604, configured to generate an information card to be displayed according to the commodity information search result, and map the information card to the virtual space scene for display.

Wherein the live-action image is obtained by live-action shooting of the robot equipment in the physical space place on the physical commodity display condition in the physical space place.

Specifically, a plurality of different subspaces are distributed in the physical space site, and the physical commodities comprise physical commodities displayed in the subspaces;

In addition, shopping guide service based on an Augmented Reality (AR) mode can be provided for a user entering the physical space place through the robot equipment:

When the information card is displayed through the robot equipment, the information card can be mapped to the position where the main body is located in the corresponding image frame for displaying.

Or the information card is displayed in a holographic projection mode.

When the information card is displayed in a holographic projection mode, the 3D model of the commodity corresponding to the information card can be displayed in the holographic projection mode.

The subject identification unit may specifically be configured to:

the information card display unit may be specifically configured to:

Wherein, the device can also include:

the reading unit is used for reading the latest position of the identified main body according to a preset period in the process that the information card moves along with the position change of the main body; wherein the time interval is related to human eye persistence time and is greater than the time interval between each frame in the image stream;

and the animation generating unit is used for generating an animation which smoothly moves from the current position to the latest position for the information card which is displayed at present after the latest position is read, keeping the position of the information card unchanged at the latest position, and generating a new animation until a new latest position is read in the next period.

The relative position relation between the information card and the anchor point of the main body is various; the position of the anchor point is determined according to the central point of the main body;

the apparatus may further include:

and the relative position relation switching unit is used for judging whether the information card collides with the edge of the screen in the current relative position relation state in the process that the information card moves along with the position change of the main body, and if so, switching to other relative position relations.

If a plurality of main bodies are identified in the image frames, respectively acquiring corresponding commodity information search results, and respectively generating information cards of corresponding types according to respective states;

at this time, the apparatus may further include:

the judging unit is used for judging whether the plurality of information cards collide with each other or not in the process of mapping the plurality of information cards into the image frame and moving along with the change of the positions of the corresponding main bodies;

and the card position moving unit is used for moving the information card display position corresponding to the lower-priority main body according to different main body state priorities corresponding to the collided information cards if the collision information cards are in the collision state.

In addition, the apparatus may further include:

and the card type switching unit is used for switching the types of the information cards mapped into the virtual space scene if the state of the main body changes in the process of mapping the information cards to the positions of the corresponding main bodies in the image frames for display.

And the judgment parameter boundaries of different main body states have buffer areas, so that when the judgment parameters change in the buffer areas, the current main body state and the types of the corresponding information cards are kept unchanged.

Wherein the state of the body comprises a display state comprising a first display state and a second display state; wherein, in the display state, the information card is used for displaying the content related to the commodity search result; the sizes and the content amounts of the information cards corresponding to the first display state and the second display state are different.

The state of the main body also comprises a discovery state, and the information card is used for displaying the identification result of the category to which the main body belongs in the discovery state.

The state of the body further comprises an interactive state;

the apparatus may further include:

and the interaction unit is used for jumping to a commodity search result list page or displaying a detail page of the commodity in the currently displayed information card after the main body enters the interaction state and keeps the threshold time.

Wherein, in case the virtual space scene is presented by a VR/mixed reality MR glasses device, the apparatus further comprises:

and the operation detection unit is used for adding the commodity corresponding to the information card into the batch settlement commodity set associated with the user when the operation of grabbing the information card and moving the information card to the target direction, which is executed by the user, is detected through the VR/MR glasses.

Corresponding to the embodiment, the embodiment of the present application further provides an information display device, and referring to fig. 7, the information display device may include:

a VR display unit 701, configured to display a pre-generated virtual space scene based on a VR technology, where the virtual space scene is generated in the following manner: performing live-action shooting on an entity in an entity space site, performing three-dimensional reconstruction on a space structure of the entity space site, splicing and matching live-action shot images into a three-dimensional reconstructed virtual space structure, and generating a virtual space scene;

a subject recognition unit 702, configured to perform subject recognition from a live-action image frame corresponding to a current display perspective in the virtual space scene;

a search unit 703 for acquiring an information search result related to the subject based on the recognized subject image;

and the information card display unit 704 is configured to generate an information card to be displayed according to the information search result, and map the information card to the virtual space scene for display.

In addition, the present application also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor implements the steps of the method described in any of the preceding method embodiments.

And an electronic device comprising:

one or more processors; and

memory associated with the one or more processors for storing program instructions which, when read and executed by the one or more processors, perform the steps of the method of any of the preceding method embodiments.

Where fig. 8 illustrates an architecture of an electronic device, for example, device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, an aircraft, etc.

Referring to fig. 8, device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and communication component 816.

The processing component 802 generally controls overall operation of the device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing element 802 may include one or more processors 820 to execute instructions to perform all or a portion of the steps of the methods provided by the disclosed solution. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operation at the device 800. Examples of such data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

A power supply component 806 provides power to the various components of the device 800. The power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 800.

The multimedia component 808 includes a screen that provides an output interface between the device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the device 800 is in an operating mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the device 800. For example, the sensor assembly 814 may detect the open/closed state of the device 800, the relative positioning of components, such as a display and keypad of the device 800, the sensor assembly 814 may also detect a change in the position of the device 800 or a component of the device 800, the presence or absence of user contact with the device 800, orientation or acceleration/deceleration of the device 800, and a change in the temperature of the device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communications component 816 is configured to facilitate communications between device 800 and other devices in a wired or wireless manner. The device 800 may access a wireless network based on a communication standard, such as WiFi, or a mobile communication network such as 2G, 3G, 4G/LTE, 5G, etc. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast associated information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communications component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described methods.

In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 804 comprising instructions, executable by the processor 820 of the device 800 to perform the methods provided by the present disclosure is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

From the above description of the embodiments, it is clear to those skilled in the art that the present application can be implemented by software plus a necessary general hardware platform. Based on such understanding, the technical solutions of the present application may be essentially or partially implemented in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method according to the embodiments or some parts of the embodiments of the present application.

All the embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, the system or system embodiments are substantially similar to the method embodiments and therefore are described in a relatively simple manner, and reference may be made to some of the descriptions of the method embodiments for related points. The above-described system and system embodiments are only illustrative, wherein the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

The method and the electronic device for providing the product search result information provided by the application are introduced in detail, and a specific example is applied in the text to explain the principle and the implementation of the application, and the description of the embodiment is only used for helping to understand the method and the core idea of the application; meanwhile, for a person skilled in the art, according to the idea of the present application, the specific embodiments and the application range may be changed. In view of the above, the description should not be taken as limiting the application.

Claims

1. A method of providing merchandise search result information, comprising:

obtaining a commodity information search result related to the identified subject;

2. The method of claim 1,

the live-action shot image is obtained by live-action shooting of the robot equipment in the physical space place on the physical commodity display condition in the physical space place.

3. The method of claim 2, further comprising:

the shopping guide service provided by the robot equipment comprises: and acquiring a real-time image stream of the display condition of the entity commodity in the entity space place, identifying a main body from the real-time image stream, acquiring a commodity information search result related to the identified main body, generating an information card to be displayed according to the commodity information search result, and displaying the information card.

4. The method of claim 3,

when the information card is displayed by the robot device, the method includes:

5. The method of claim 1,

the subject recognition from the live-action image frame in the virtual space scene comprises:

the information card to be displayed is generated according to the commodity information search result, and the method comprises the following steps:

6. The method according to claim 4 or 5,

7. The method according to claim 4 or 5,

the method further comprises the following steps:

8. The method according to claim 4 or 5,

the method further comprises the following steps:

if so, the information card display position corresponding to the lower-priority main body is moved according to the different main body state priorities corresponding to the collided information cards.

9. The method of claim 4 or 5, further comprising:

10. The method of claim 4 or 5,

the state of the main body comprises a display state, and the display state comprises a first display state and a second display state; wherein, in the display state, the information card is used for displaying the content related to the commodity search result; the sizes and the content amounts of the information cards corresponding to the first display state and the second display state are different.

11. A robot apparatus is characterized in that,

the robot device is applied to a physical space place;

12. An information display method, comprising:

13. A computer-readable storage medium, on which a computer program is stored, which program, when being executed by a processor, is adapted to carry out the steps of the method of one of the claims 1 to 10, 12.

14. An electronic device, comprising:

one or more processors; and

memory associated with the one or more processors for storing program instructions which, when read and executed by the one or more processors, perform the steps of the method of any one of claims 1 to 10, 12.