CN112055034A

CN112055034A - Interaction method and system based on optical communication device

Info

Publication number: CN112055034A
Application number: CN201910485776.XA
Authority: CN
Inventors: 李江亮; 方俊
Original assignee: Beijing Whyhow Information Technology Co Ltd
Current assignee: Beijing Whyhow Information Technology Co Ltd
Priority date: 2019-06-05
Filing date: 2019-06-05
Publication date: 2020-12-08
Anticipated expiration: 2039-06-05
Also published as: CN112055034B

Abstract

An interaction method and system based on an optical communication device, the method comprising: the server receives request information from the first device and information related to the location of the first device; the server obtains the position information of the first equipment relative to the optical communication device through the information related to the position of the first equipment; the server sets a virtual object having spatial position information associated with the first device; the server provides the request information to the second device; the server receives a response of the second equipment to the request information; and the server sends information related to the virtual object to the second device, the information including spatial position information of the virtual object, wherein the information related to the virtual object is usable by the second device to render the virtual object on its display medium based on its position information and attitude information relative to the optical communication apparatus.

Description

Interaction method and system based on optical communication device

Technical Field

The invention belongs to the field of information interaction, and particularly relates to an interaction method and system based on an optical communication device.

Background

The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

With the wide popularization of the internet, various industries try to develop a new service providing mode by using an internet platform, and the internet + becomes a research hotspot. One widely used service approach is location-based service, and in many application scenarios, one of the users or service providers must know the exact location of the other to enable convenient service interaction. However, the prior art generally uses GPS signals to provide location information, which is not so accurate that the problem is not well solved in many cases.

In addition, the location-based service in the prior art is usually based on a "service provider + user" model, wherein the service provider provides services, such as express delivery service, meal ordering service, taxi service, etc., to the user based on the user's location information. Solutions that enable location-based services between different individual users are lacking in the prior art.

In order to solve the above problems, the present application provides an interaction method and system based on an optical communication device.

Disclosure of Invention

One aspect of the present invention relates to an interaction method based on an optical communication device, including: a server receives request information from a first device and information related to a location of the first device; the server obtains the position information of the first equipment relative to the optical communication device through the information related to the position of the first equipment; the server sets a virtual object having spatial position information associated with the first device, wherein the spatial position information of the virtual object is determined based on the position information of the first device relative to an optical communication apparatus; the server provides the request information to the second device; the server receives a response of the second device to the request information; and the server sending information related to the virtual object to a second device, the information including spatial position information of the virtual object, wherein the information related to the virtual object is usable by the second device to render the virtual object on its display medium based on its position information and attitude information relative to the optical communication apparatus.

Another aspect of the present invention relates to an interactive system based on an optical communication apparatus, comprising: one or more optical communication devices; and a server configured to implement the above method.

Yet another aspect of the present invention relates to an interaction method based on an optical communication apparatus, including: the second device receives request information from the first device from the server; the second device sends a response to the request information to the server; the second device receiving information related to a virtual object from the server, the information including spatial location information of the virtual object, the spatial location information being determined based on location information of the first device relative to an optical communication apparatus; the second device determining its position information and attitude information relative to the optical communication apparatus; and the second device renders the virtual object on its display medium based on its position information and pose information relative to the optical communication apparatus and the information related to the virtual object.

Yet another aspect of the invention relates to an apparatus for interaction based on optical communication means, configured to implement the method described above.

A further aspect of the invention relates to a storage medium in which a computer program is stored which, when being executed by a processor, can be used for carrying out the above-mentioned method.

Yet another aspect of the invention relates to an electronic device comprising a processor and a memory, in which a computer program is stored which, when being executed by the processor, is operative to carry out the method as described above.

Drawings

Embodiments of the invention are further described below with reference to the accompanying drawings, in which:

FIG. 1 illustrates an exemplary optical label;

FIG. 2 illustrates an exemplary optical label network;

fig. 3 shows a light label placed over one of the hoistway doors;

FIG. 4 shows a schematic diagram of a clerk dispensing coffee to a user;

FIG. 5 illustrates a schematic diagram of superimposing a virtual object on a display medium of a clerk's device;

FIG. 6 illustrates an optical label-based interaction method according to one embodiment;

FIG. 7 illustrates an optical label-based interaction method according to one embodiment;

FIG. 8 illustrates an optical label-based interaction method according to one embodiment;

FIG. 9 shows an interactive system comprising two optical labels;

FIG. 10 illustrates an application scenario that enables a location-based service scheme between different individual users;

FIG. 11 is a schematic diagram illustrating the superimposition of a virtual object on a display medium of a second device; and

FIG. 12 illustrates an optical label-based interaction method according to one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail by embodiments with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

Optical communication devices are also referred to as optical labels, and these two terms are used interchangeably herein. The optical label can transmit information by emitting different lights, which has advantages of a long recognition distance, a loose requirement on visible light conditions, and the information transmitted by the optical label can be changed with time, so that a large information capacity and a flexible configuration capability can be provided.

An optical label may typically include a controller and at least one light source, the controller may drive the light source through different driving modes to communicate different information to the outside. Fig. 1 shows an exemplary optical label 100 comprising three light sources (first light source 101, second light source 102, third light source 103, respectively). Optical label 100 further comprises a controller (not shown in fig. 1) for selecting a respective driving mode for each light source in dependence on the information to be communicated. For example, in different driving modes, the controller may control the manner in which the light source emits light using different driving signals, such that when the optical label 100 is photographed using the imaging-capable device, the image of the light source therein may take on different appearances (e.g., different colors, patterns, brightness, etc.). By analyzing the imaging of the light sources in the optical label 100, the driving pattern of each light source at the moment can be analyzed, so that the information transmitted by the optical label 100 at the moment can be analyzed.

In order to provide corresponding services to subscribers based on optical labels, each optical label may be assigned identification Information (ID) for uniquely identifying or identifying the optical label by a manufacturer, manager, user, or the like of the optical label. In general, the light source may be driven by a controller in the optical tag to transmit the identification information outwards, and a user may use the device to perform image capture on the optical tag to obtain the identification information transmitted by the optical tag, so that a corresponding service may be accessed based on the identification information, for example, accessing a web page associated with the identification information of the optical tag, acquiring other information associated with the identification information (e.g., location information of the optical tag corresponding to the identification information), and so on. The devices referred to herein may be, for example, mobile devices that a user carries with him (e.g., a cell phone with a camera, a tablet, smart glasses, a smart helmet, a smart watch, etc.), or machines that are capable of autonomous movement (e.g., a drone, an unmanned automobile, a robot, etc.). The device can acquire a plurality of images containing the optical label by continuously acquiring images of the optical label through the camera on the device, and analyzes the image of the optical label (or each light source in the optical label) in each image through a built-in application program to identify the information transmitted by the optical label.

The optical label may be installed at a fixed location and may store identification Information (ID) of the optical label and any other information (e.g., location information) in the server. In reality, a large number of optical labels may be constructed into an optical label network. FIG. 2 illustrates an exemplary optical label network that includes a plurality of optical labels and at least one server, wherein information associated with each optical label may be stored on the server. For example, identification Information (ID) or any other information of each optical label, such as service information related to the optical label, description information or attributes related to the optical label, such as position information, physical size information, physical shape information, pose or orientation information, etc. of the optical label may be maintained on the server. The device may use the identification information of the identified optical label to obtain further information related to the optical label from the server query. The position information of the optical label may refer to an actual position of the optical label in the physical world, which may be indicated by geographical coordinate information. A server may be a software program running on a computing device, or a cluster of computing devices. The optical label may be offline, i.e., the optical label does not need to communicate with the server. Of course, it will be appreciated that an online optical tag capable of communicating with a server is also possible.

Fig. 3 shows a light label placed over one of the hoistway doors. When a user scans the optical label using the device, the identification information conveyed by the optical label can be identified, and the corresponding service can be accessed using the identification information, for example, accessing a web page of the restaurant associated with the identification information of the optical label. Optical labels can be deployed in various places, e.g., on squares, store fronts, in restaurants, as desired.

The optical labels may be used as anchor points to enable the superimposition of virtual objects onto the real scene, for example, to accurately identify where a user or device is located in the real scene using the virtual objects. The virtual object may be, for example, an icon, a picture, text, a number, an emoticon, a virtual three-dimensional object, a three-dimensional scene model, a piece of animation, a piece of video, and so forth.

To illustrate with an instant courier service, a user carrying a device (e.g., a cell phone, smart glasses, etc.) may want to purchase a cup of coffee while walking on a mall and wait in place for a clerk at the coffee shop to send the coffee to his location. The user may use his device to scan and identify an optical label on a floor of a coffee shop disposed around him, and access a corresponding service to purchase a cup of coffee through the identified optical label identification information. As the user scans the optical label using his device, an image of the optical label may be taken and a relative positioning may be performed by analyzing the image to determine the position information of the user (more precisely, the user's device) relative to the optical label, which may be sent to the server of the cafe together with the coffee purchase request. The server of the coffee shop may set a virtual object, which may be, for example, an order number "123" corresponding to a coffee purchase request, after receiving the coffee purchase request of the user. The server may also determine spatial location information of the virtual object according to the received location information of the user equipment relative to the optical label, for example, the location of the virtual object may be set to be at the location of the user equipment or 1 meter above the location of the user equipment. After the coffee shop has prepared the user's coffee, the coffee shop clerk may proceed with the coffee delivery. Fig. 4 shows a schematic diagram of a clerk dispensing coffee to a user. During the distribution process, the clerk can scan the optical label using his device (e.g., cell phone, smart glasses, etc.) to determine the location information and pose information of the clerk's device relative to the optical label. The server may send information about the virtual object (including its spatial location information) to the clerk's device when the clerk scans the optical label using his device or at other times. As such, the positional relationship of the virtual object relative to the clerk device can be determined with the optical label as an intermediate anchor point, and the virtual object (e.g., the sequence of numbers "123") can be rendered at an appropriate location on the display medium of the clerk device based further on the pose of the clerk device. For example, a digital sequence "123" may be superimposed at a suitable position in a real scene shown on a display screen of a clerk device, where the position of the digital sequence "123" or a position about 1 meter below the position is the position of the coffee purchaser. FIG. 5 illustrates a schematic diagram of superimposing a virtual object on a display medium of a clerk's device. In this way, the optical label can be used as an anchor point to realize accurate superposition of the virtual object in a real scene, thereby helping a cafe clerk to quickly find the position where a coffee purchaser is located so as to realize delivery of coffee. Preferably, the cafe clerk can use smart glasses instead of a cell phone during the distribution process to achieve more convenient delivery.

The restaurant meal delivery service is taken as an example for explanation, when a user with equipment has a meal in a restaurant, the user can use the equipment to scan and identify a certain optical label arranged in the restaurant, and access the corresponding meal ordering service through the identified optical label identification information. As the user scans the optical label using his device, an image of the optical label may be taken and a relative positioning may be performed by analyzing the image to determine the position information of the user (more precisely, the user's device) relative to the optical label, which may be sent to the server of the restaurant along with the order request. The restaurant's server, upon receiving the user's order request, may set a virtual object, which may be, for example, order number "456" corresponding to the order request. The server may also determine spatial location information of the virtual object according to the received location information of the user equipment relative to the optical label, for example, the location of the virtual object may be set to be at the location of the user equipment or 1 meter above the location of the user equipment. After the restaurant has prepared the user's dishes, the restaurant clerk may make the dish distribution. During the distribution process, the restaurant clerk can scan the optical label using their device (e.g., cell phone, smart glasses, etc.) to determine the location information and pose information of the clerk's device relative to the optical label. The server may send information about the virtual object (including its spatial location information) to the clerk's device when the clerk scans the optical label using his device or at other times. As such, the positional relationship of the virtual object relative to the clerk device can be determined with the optical label as an intermediate anchor point, and the virtual object can be rendered at an appropriate location (e.g., the numerical sequence "456") on the display medium of the clerk device further based on the pose of the clerk device. For example, a digital sequence "456" may be superimposed at a suitable position in a real scene displayed on a display screen of a clerk device, where the position of the digital sequence "456" or a position about 1 meter below the position is the position of a user to whom a dish needs to be delivered. In this way, the optical label can be used as an anchor point to realize accurate superposition of the virtual object in a real scene, so that restaurant employees can be helped to quickly find the positions of the ordering persons. Preferably, restaurant store personnel may use smart glasses instead of cell phones to enable more convenient delivery. In one embodiment, the user may not order by scanning and recognizing the optical label, but may order in any other manner. Instead of determining the position information of the user relative to the optical label by capturing an image of the optical label, the user may scan the two-dimensional code on the table or directly send the table number to the restaurant server to inform the restaurant server of the position of the user, and the restaurant server may store the relative positional relationship between each table and the optical label in advance and determine the position information of the user relative to the optical label based on the identification information of the two-dimensional code scanned by the user or the table number sent by the user.

FIG. 6 illustrates an optical label-based interaction method according to one embodiment, the method comprising the steps of:

step 601: the server receives information from the first device, the information including location information of the first device relative to the optical label.

The information from the first device may be product purchase information issued by the user of the first device to the server, but may be any other information. When the first device transmits its position information with respect to the optical label to the server, it may transmit the identification information of the optical label, which it recognizes by scanning the optical label, together.

The device may determine its position information relative to the optical label in various ways, which may include distance information and direction information of the device relative to the optical label. In one embodiment, the device may determine its position information relative to the optical label by capturing an image that includes the optical label and analyzing the image. For example, the device may determine the relative distance of the optical label from the device (the greater the imaging, the closer the distance; the smaller the imaging, the further the distance) by the imaging size of the optical label in the image and optionally other information (e.g., actual physical dimension information of the optical label, the focal length of the camera of the device). The device may obtain actual physical size information of the optical label from the server using the identification information of the optical label, or the optical label may have a uniform physical size and store the physical size on the device. The device may determine orientation information of the device relative to the optical label by perspective distortion of the optical label imaging in the image including the optical label and optionally other information (e.g., imaging location of the optical label). The device may obtain physical shape information of the optical label from a server using identification information of the optical label, or the optical label may have a uniform physical shape and store the physical shape on the device. In one embodiment, the device may also directly obtain the relative distance of the optical label from the device through a depth camera or a binocular camera or the like mounted thereon. The device may also use any other positioning method known in the art to determine its position information relative to the optical label.

Step 602: the server sets a virtual object associated with the first device having spatial location information, wherein the spatial location information of the virtual object is determined based on the location information of the first device relative to the optical label.

Upon receiving information from the first device (e.g., product purchase information), the server may set up a virtual object associated with the first device. The virtual object may be, for example, an order number corresponding to product purchase information issued by the first device, a name of a user who purchased the product, identification information of an item to be delivered, a simple virtual icon, and so on. The spatial position information of the virtual object is determined from the position information of the first device relative to the optical label, which is preferably also the position information relative to the optical label. The spatial position of the virtual object may simply be determined as the position of the first device, or may be determined as another position, for example, another position located near the position of the first device.

Step 603: the server sends information related to the virtual object to the second device, wherein the information comprises spatial position information of the virtual object.

The information on the virtual object is related information for describing the virtual object, and may include, for example, a picture, a letter, a number, an icon, and the like included in the virtual object, and may also include shape information, color information, size information, posture information, and the like of the virtual object. Based on this information, the device may render the corresponding virtual object. The information about the virtual object includes spatial position information of the virtual object, which may be position information with respect to the optical label (e.g., distance information of the virtual object with respect to the optical label and direction information with respect to the optical label). In one embodiment, the information related to the virtual object may further include superimposed pose information of the virtual object, which may be pose information of the virtual object with respect to the optical label, or its pose information in a real world coordinate system.

In one embodiment, the server may send the information relating to the virtual object directly to the second device, for example over a wireless link. In another embodiment, the second device may identify the identification information conveyed by the optical label when scanning the optical label and use the identification information of the optical label to obtain information about the virtual object from the server.

Step 604: the second device renders the virtual object on a display medium of the second device based on its position information and pose information relative to the optical label and information related to the virtual object.

The second device may determine its position information relative to the optical label in various ways, such as the various ways described above in step 601, which are not described herein again.

The second device may determine its pose information, which may be used to determine the extent or boundaries of the real scene captured by the device. Typically, the pose information of the device is actually pose information of an image capture device (e.g., a camera) of the device. In one embodiment, the second device may scan the optical label and may determine its pose information relative to the optical label based on the imaging of the optical label, and may consider the second device as currently facing the optical label when the imaging location or imaging area of the optical label is centered in the imaging field of view of the second device. The direction of imaging of the optical label may further be taken into account when determining the pose of the second device. As the pose of the second device changes, the imaging position and/or imaging direction of the optical label on the second device changes accordingly, and therefore pose information of the second device relative to the optical label can be obtained according to the imaging of the optical label on the second device.

In one embodiment, position information and pose information (which may collectively be referred to as pose information) of the device relative to the optical labels may also be determined as follows. In particular, a coordinate system may be established from the optical label, which may be referred to as the optical label coordinate system. Some points on the optical label may be determined as some spatial points in the optical label coordinate system, and the coordinates of these spatial points in the optical label coordinate system may be determined according to the physical size information and/or the physical shape information of the optical label. Some of the points on the optical label may be, for example, corners of a housing of the optical label, ends of a light source in the optical label, some identification points in the optical label, and so on. According to the object structure features or the geometric structure features of the optical label, image points corresponding to the space points can be found in the image shot by the equipment camera, and the positions of the image points in the image are determined. According to the coordinates of each space point in the optical label coordinate system and the positions of corresponding image points in the image, and by combining the internal reference information of the equipment camera, the pose information (R, t) of the equipment camera in the optical label coordinate system when the image is shot can be obtained through calculation, wherein R is a rotation matrix which can be used for representing the pose information of the equipment camera in the optical label coordinate system, and t is a displacement vector which can be used for representing the position information of the equipment camera in the optical label coordinate system. Methods of calculating R, t are known in the art, and R, t may be calculated using, for example, the 3D-2D PnP (Passive-n-Point) method, and will not be described in detail herein in order not to obscure the invention. The rotation matrix R and the displacement vector t may actually describe how the coordinates of a certain point are transformed between the optical label coordinate system and the device camera coordinate system. For example, by rotating the matrix R and the displacement vector t, the coordinates of a certain point in the optical label coordinate system can be converted into coordinates in the device camera coordinate system and can further be converted into the position of an image point in the image. In this way, for a virtual object having a plurality of feature points (a plurality of points on the outline of the virtual object), the coordinates of the plurality of feature points in the light tag coordinate system (i.e., the position information relative to the light tag) may be included in the spatial position information of the virtual object, and based on the coordinates of the plurality of feature points in the light tag coordinate system, the coordinates of the plurality of feature points in the device camera coordinate system may be determined, so that the respective imaging positions of the plurality of feature points on the device may be determined. Once the respective imaging positions of the plurality of feature points of the virtual object are determined, the position, size, or posture of imaging of the entire virtual object can be determined accordingly.

After the spatial position information of the virtual object and the position information of the second device relative to the optical label are obtained through the above steps, a three-dimensional spatial coordinate system with the optical label as an origin can be actually created, wherein the second device and the virtual object to be superimposed each have accurate spatial coordinates in the coordinate system. In one embodiment, the position information of the virtual object to be superimposed with respect to the second device may also be determined based on the spatial position information of the virtual object and the position information of the second device with respect to the optical label. On the basis of the above, the virtual object may be superimposed in the real scene based on the pose information of the second device. For example, the imaging size of the virtual object to be superimposed may be determined based on the relative distance of the second device and the virtual object to be superimposed, and the imaging position of the virtual object to be superimposed on the display medium of the second device may be determined based on the relative direction of the second device and the virtual object to be superimposed and the posture information of the second device. In one embodiment, the virtual object to be superimposed may have a default imaging size, in which case only the imaging position of the virtual object to be superimposed on the display medium of the second device may be determined without determining its imaging size. In the case where the superimposition information includes superimposition attitude information of the virtual object, the attitude of the superimposed virtual object may be further determined. In one embodiment, the position, size, or pose, etc., of the imaging of the virtual object to be superimposed on the device may be determined from the pose information (R, t) of the device (more precisely, the camera of the device) with respect to the light labels calculated above. In one case, if it is determined that the virtual object to be superimposed is not currently in the field of view of the second device (e.g., the imaging position of the virtual object is outside the display screen), the virtual object is not displayed.

In some cases, the virtual object may be located at a position that is not in the field of view of the second device (e.g., the field of view of the camera of the second device) when the second device scans the optical label, in which case, an attempt may be made to translate and/or rotate the second device and track changes in the pose of the second device via sensors (e.g., accelerometer, gyroscope, visual odometer, etc.) built into the second device to determine a new field of view of the second device. When the virtual object is positioned in the field of view of the second device, the virtual object may be presented on a display medium of the second device. The technology for tracking the posture change of the equipment by using the built-in sensor of the equipment is a well-known technology in the field of augmented reality and virtual reality, and is not described in detail herein.

In the above embodiments, the optical label is actually used as an anchor point, based on which an accurate overlay of the virtual object in the real scene observed by the second device is achieved. The device may present the real scene in a variety of possible ways. For example, the device may capture real world information via a camera and use the information to render the real world scene on a display screen on which an image of a virtual object may be superimposed. The device (e.g., smart glasses) may also reproduce the real scene not through the display screen, but simply through a prism, lens, mirror, transparent object (e.g., glass), etc., into which the image of the virtual object may be optically superimposed. The above-described display screen, prism, lens, mirror, transparent object, etc. may be collectively referred to as a display medium of the device on which the virtual object may be presented. For example, in one type of optical see-through augmented reality device, a user observes a real scene through a particular lens, while the lens may reflect an image of a virtual object into the user's eyes. In one embodiment, a user of the device may directly observe a real scene or part thereof, which does not need to be reproduced via any medium before being observed by the user's eyes, and virtual objects may be optically superimposed into the real scene. Thus, a real scene or a portion thereof does not necessarily need to be rendered or reproduced by the device before being observed by the eyes of the user.

After superimposing the virtual object, the device may be translated and/or rotated, in which case the change in position and the change in attitude may be measured using methods known in the art (e.g., using an acceleration sensor, gyroscope, visual odometer, etc. built into the device) to adjust the display of the virtual object, e.g., to change its imaging position, imaging size, viewing angle, virtual object into the device field of view, virtual object out of the device field of view, etc. This is known in the art and will not be described in further detail. However, due to the accuracy of the built-in sensor of the device and the lack of texture features in some scenes (for example, dark night with poor light, white wall without texture features, blue sky, etc.), the method of tracking the position and the posture of the device through the built-in sensor or the visual odometer adopted by the prior art is easy to cause the drift of the superposed virtual objects. For example, after a period of translation and/or rotation of the device, when the virtual object appears in the field of view again, it is found that its current overlay position deviates from the initial overlay position. In one embodiment, the device may re-determine its position information relative to the optical tag and its pose information relative to the optical tag (e.g., when the optical tag re-enters the device field of view after leaving the device field of view, or at regular intervals while the optical tag remains in the device field of view), and re-determine the imaging position and/or imaging size of the virtual object based on the overlay position information of the virtual object and the position information and pose information of the device relative to the optical tag, thereby correcting the overlay of the virtual object in the real scene. For example, if the imaging position or imaging size of the virtual object currently displayed by the device differs from the re-determined imaging position or imaging size or the difference exceeds a preset threshold, the device may superimpose the virtual object according to the re-determined imaging position and imaging size. In this way, the position of the superimposed virtual object can be prevented from drifting along with the rotation or movement of the device.

In one embodiment, after superimposing a virtual object, the device or its user may perform an operation on the virtual object to change the properties of the virtual object. For example, the device or its user may move the position of the virtual object, change the pose of the virtual object, change the size or color of the virtual object, add annotations to the virtual object, and so forth. In one embodiment, after the device or its user changes the properties of the virtual object, the modified property information of the virtual object may be uploaded to the server. The server may update the relevant information of the virtual object it stores based on the modified attribute information. In one embodiment, the device or its user may perform a delete operation on the overlaid virtual object and notify the server. Still taking the coffee purchasing application mentioned above as an example, the virtual number sequence "123" associated with the user may be deleted when the cafe clerk carrying the second device completes the coffee delivery to the user.

In other embodiments, the information from the first device may not include the location information of the first device relative to the optical label, and the server may obtain the location information of the first device relative to the optical label in other manners. In one embodiment, the server may obtain the location information of the first device relative to the optical label by analyzing information from the first device. For example, the information from the first device may include an image taken by the first device containing an optical label, and the server may obtain the location information of the first device relative to the optical label by analyzing the image. In one embodiment, the server may use information from the first device to obtain location information of the first device relative to the optical label through a query. For example, the information from the first device may be two-dimensional code identification information or identification information such as a table number, based on which the server may obtain the position information of the first device with respect to the optical label by querying. In one embodiment, the server may obtain the location information (e.g., absolute location information) of the first device from the information from the first device, and obtain the location information of the first device relative to the optical label from the location information and the location information of the optical label. The position information of the first device may be, for example, its GPS position information, which, although the accuracy of the current GPS position information is not very high, may also be applicable in some application scenarios where the accuracy requirement for the virtual object overlay is not high. Any information that can be used to obtain the location of the device (e.g., an image taken by the device containing an optical label, two-dimensional code identification information scanned by the device, a table number transmitted by the device, etc.) can be referred to as information relating to the location of the device.

In one embodiment, multiple virtual objects may be presented simultaneously on the display medium of the second device. In one embodiment, the server may determine one or more virtual objects that need to be rendered on the display medium of the second device. For example, if a first clerk at a coffee shop is to send coffee for a first party, the server can send information about the virtual object associated with the first user to the device of the first clerk; additionally, if a second clerk at the coffee shop wants to send coffee for both the second user and a third user, the server can send information about the virtual object associated with the second user and information about the virtual object associated with the third user to the device of the second clerk. In the case where there are a plurality of virtual objects, overlapping, blocking, and the like may occur when the virtual objects are superimposed. In one embodiment, overlapping, occlusion, etc. situations between virtual objects may be considered when overlaying multiple virtual objects, and only un-occluded virtual objects or un-occluded portions of virtual objects are overlaid or rendered in the real scene. In another embodiment, it is also contemplated to set virtual objects or portions thereof that occlude other virtual objects to be semi-transparent and also overlay or render the occluded virtual objects or portions thereof, thereby enabling the device user to view all of the virtual objects.

In some cases, the user may change his location after having sent his location information relative to the optical label to the server using the first device. For example, a user who has purchased coffee may move around after sending a purchase request together with its position information with respect to the optical label. In order to enable the server to know the latest location of the user or his first device in time, new location information of the first device may be sent to the server. The first device may determine its latest position information relative to the optical label in the various ways mentioned above (e.g., by capturing an image including the optical label and analyzing the image), and may also track changes in the position of the first device through sensors built into the first device (e.g., acceleration sensors, gyroscopes, etc.). The new location information of the first device may be periodically transmitted to the server, or the transmission of the new location information may be initiated when the difference between the new location of the first device and the location last transmitted to the server is greater than some preset threshold. In this way, the server can know the new position information of the first device in time, and can update the spatial position information of the virtual object accordingly and notify the second device of the new spatial position information of the virtual object. The second device may accordingly use the new spatial position information of the virtual object to render or update the virtual object on its display medium.

Fig. 7 shows an interaction method based on an optical label according to an embodiment, which can implement tracking of the location of the first device, where

steps

701 and 704 are similar to

steps

601 and 604 of fig. 6, and are not described herein again. The interaction method of fig. 7 further comprises the steps of:

step 705: the server receives new information from the first device.

The new information may be any information that can be used to obtain the position of the first device relative to the optical label, including displacement information of the first device obtained by tracking by a sensor built into the first device.

Step 706: the server updates the position information of the first device relative to the optical label based on the new information.

Step 707: the server updates the spatial position information of the virtual object based on the updated position information of the first device relative to the optical label.

Step 708: the server sends the updated spatial position information of the virtual object to the second device so that the second device can render or update the virtual object on its display medium based on its position information and attitude information relative to the optical label and the updated spatial position information of the virtual object.

In one embodiment, a virtual object associated with the second device may also be enabled to be presented on a display medium of the first device. Taking the coffee purchase service described above as an example, during the distribution process, the clerk can scan the optical label using his device (e.g., cell phone, smart glasses, etc.) to determine the location information and pose information of the clerk's device relative to the optical label. After this, the clerk's device can send its location information relative to the optical label to the server. The server may set a virtual object for the clerk's device, the spatial location information of the virtual object being determined based on the location information of the clerk's device relative to the optical label. The server may send information about the virtual object to the device of the user who purchased the coffee and may inform the user that his coffee is being delivered. The user may then scan the optical label using their device (e.g., cell phone, smart glasses, etc.) to determine position information and pose information of the user device relative to the optical label. Thus, the user device may render a virtual object (e.g., the number sequence "123") at an appropriate location on the display medium of the user device based on its position and pose information relative to the optical label and the relevant information of the virtual object associated with the store clerk device, which facilitates more convenient interaction between the user and the store clerk. Since the clerk who delivers the coffee is typically in the middle of a movement, the location of the clerk device can be tracked and sent to the server periodically or in real-time to update the spatial location information of the virtual object associated with the clerk device, which is then sent to the user's device.

Fig. 8 shows an interaction method based on optical labels according to an embodiment, which may further present a virtual object associated with a second device on a display medium of a first device, where

steps

801 and 804 are similar to step 601 and 604 of fig. 6 and are not described herein again. The interaction method of fig. 8 further comprises the steps of:

step 805: the server receives information from the second device and determines location information of the second device relative to the optical label.

Step 806: the server sets another virtual object having spatial location information associated with the second device, wherein the spatial location information of the other virtual object is determined based on the location information of the second device relative to the optical label.

Step 807: the server sends information related to the other virtual object to the first device, the information including spatial position information of the other virtual object, so that the first device can present the other virtual object on a display medium of the first device based on the position information and the posture information of the first device relative to the optical label and the information related to the other virtual object.

In one embodiment, in the method shown in fig. 8, the position information of the second device and the spatial position information of the other virtual object may be further updated in a manner similar to the method of fig. 7, so that the other virtual object presented on the display medium of the first device can track the position of the second device.

In many scenarios, there may be more than one optical label, but rather an optical label network as shown in fig. 2, where the server may know the location information of the individual optical labels or the relative location relationship between them. In these scenarios, the optical labels scanned by the first device and the second device may not be the same optical label, the first device may scan a plurality of different optical labels at different times to provide or update its location information (providing or updating the location information may send identification information of the associated optical label), and the second device may scan a plurality of different optical labels at different times to determine its location information and pose information. For example, as shown in fig. 9, a plurality of optical labels including a first optical label and a second optical label may be arranged in one restaurant, and a server or a second device of a restaurant clerk can know the relative positional relationship of the first optical label and the second optical label. A dining user may scan a first optical label using a first device to determine their position relative to the first optical label, and a restaurant clerk may scan a second optical label using their second device to determine position information and pose information of the second device relative to the second optical label while delivering dishes. Since the relative positional relationship of the first optical label and the second optical label is known, positional information of the first device relative to the first optical label may be converted into positional information relative to the second optical label, or spatial positional information of a virtual object associated with the first device relative to the first optical label may be converted into spatial positional information relative to the second optical label, thereby enabling accurate rendering of the virtual object on the second device.

In some scenarios, the first device and the second device may initially be relatively far apart, in which case a user of the second device may first travel to the vicinity of the first device using some existing navigation means (e.g., GPS navigation), and then present a virtual object associated with the first device on a display medium of the second device by scanning surrounding light tags using the second device.

Fig. 10 illustrates an application scenario that enables a location-based service scheme between different individual users. The scene has an optical label, and a first user and a second user located near the optical label, where the first user carries a first device, and the second user carries a second device, and the first device and the second device may be, for example, a mobile phone, smart glasses, and the like. The optical label shown in fig. 10 may be arranged on a square, for example. A first user on a square finds himself temporarily needing to use something (e.g. scissors, tape, medicine, etc.) but who is not carrying him with him. To do so, the first user may scan and identify the optical label using the first device to send a request, e.g., "borrow or buy item a," to a server associated with the optical label. While the first user scans the optical label using the first device, an image of the optical label may be taken and relative positioning may be performed based on the image to determine the position information of the first user (more precisely, the first device of the first user) relative to the optical label, which may be sent to the server together with the request. The server may set a virtual object for the first device after receiving the request from the first device, the virtual object may be, for example, an indication arrow. The server may also determine spatial location information of the virtual object according to the received location information of the first device relative to the optical label, for example, the location of the virtual object may be set to be the location of the first device or 1 meter above the location of the first device. After the first device sends a request to the server, the second user may scan and identify the optical label using the second device and receive the request sent by the first device from the server associated with the optical label. If the second user is able to provide item A to satisfy the first user's request, it may send a response to the server using the second device. The server, upon receiving the response from the second device, may transmit the related information of the virtual object (including the spatial position information thereof) previously set for the first device to the second device. The second device may determine its position information and pose information relative to the optical label as it scans the optical label. As such, the positional relationship of the virtual object relative to the second device may be determined with the optical label as an anchor point, and the virtual object may be rendered at a suitable location (e.g., indicative of an arrow) on the display medium of the second device further based on the pose of the second device. For example, an indication arrow may be superimposed at a suitable position in a real scene shown on a display screen of the second device, where the position of the indication arrow or a position 1 meter or so below the indication arrow is the position of the first user or the first device. FIG. 11 shows a schematic diagram of superimposing a virtual object (e.g., a pointing arrow) on a display medium of a second device. In this way, the optical label can be used as an anchor point to realize accurate superposition of the virtual object, thereby helping the second user to quickly find the position where the first user is located so as to realize interaction between the two users.

FIG. 12 shows an optical label-based interaction method according to one embodiment, the method comprising the steps of:

step 1201: the server receives the request information from the first device and the location information of the first device relative to the optical label.

The requested information from the first device may be any requested information that the user of the first device makes to the server, such as a help request, a friend-making request, an appointment request, a notification request (e.g., the user of the first device is performing somewhere on site), and so forth. When the first device transmits information to the server, the first device may transmit identification information of the optical label, which is recognized by scanning the optical label, together. The device may determine its position information relative to the optical label in the various ways described above.

Step 1202: the server sets a virtual object associated with the first device having spatial location information, wherein the spatial location information of the virtual object is determined based on the location information of the first device relative to the optical label.

Upon receiving the request information from the first device, the server may set a virtual object associated with the first device. The spatial position information of the virtual object is determined from the position information of the first device relative to the optical label, which is preferably also the position information relative to the optical label.

Step 1203: the server provides the request information to the second device.

When the second device scans and identifies the optical label, it may interact with the server through the identification information of the identified optical label. During the interaction, the server may send the request information it receives from the first device to the second device. For example, the server may send "the first user borrows or buys item a" to the second device. The requested information from the first device may be presented to the second user on the second device in various forms, such as in the form of a short message, in the form of a pop-up notification in an application, and so forth. Preferably, a virtual message board may be presented on or near the optical label image presented on the display medium of the second device and the requested information from the first device displayed thereon. It will be appreciated that the server may send multiple pieces of requested information sent by multiple different devices to the second device so that a user of the second device may view the requested information and decide which requested information to respond to. The server, when providing the requested information to the second device, may provide other information related to the requested information, such as the location of the user who initiated the requested information, the time at which the request was initiated, and so forth.

Step 1204: the server receives a response of the second device to the request information.

If the second user is interested in the requested information from the first device (e.g., the second user can satisfy the help request sent by the first user through the first device), it may send a response to the server through the second device. In one embodiment, after the server receives a response from the second device, it may no longer send the requested information from the first device to the other devices.

Step 1205: the server sends information related to the virtual object to the second device, wherein the information comprises spatial position information of the virtual object.

Upon receiving the response from the second device, the server may send information related to the virtual object to the second device. The information about the virtual object includes spatial position information of the virtual object, which may be position information with respect to the optical label. In one embodiment, the information related to the virtual object may further include overlay pose information of the virtual object.

Step 1206: the second device renders the virtual object on a display medium of the second device based on its position information and pose information relative to the optical label and information related to the virtual object.

The second device may determine its position information and attitude information relative to the optical label in various ways described above, and will not be described in detail here.

In some embodiments, the information from the first device may also not include the location information of the first device relative to the optical label, and the server may obtain the location information of the first device relative to the optical label in the other various ways mentioned above. For example, information about the location of the first device may be included in the information from the first device, and the server may obtain location information of the first device relative to the optical label from the information about the location of the first device.

In some embodiments, the method shown in fig. 12 may further track the location of the first device in a manner similar to the method shown in fig. 7. In some embodiments, the method illustrated in FIG. 12 may further present a virtual object associated with the second device on a display medium of the first device in a manner similar to the method illustrated in FIG. 8.

In some embodiments, the request information from the first device may have an associated validity time range to define within which time range the request information is valid. For example, the first user may set the request to be valid for a certain time (e.g., 10 minutes) when sending the help request through the first device. After exceeding the validity time range, the server may no longer transmit the requested information from the first device to the other device.

In some embodiments, the requested information from the first device may have an associated geographic range of validity to define within which geographic range the requested information is valid. For example, in one application scenario, a first user sends an event notification request to a server via a first device (e.g., the first user is performing live at location a) to invite other users to watch. The first user may set a geographic range associated with the event notification. For example, a first user may set that the event notification can be received by other users within 500 meters of their surroundings, the first user may also set that the event notification can be received by other users interacting with optical labels within 500 meters of their surroundings, and so on. As such, the server may determine to which users' devices to send the event notification based on the determined relative distances between the other users and the first user, or the determined relative distances between different optical labels and the first user.

The device referred to herein may be a device carried by a user (e.g., a cell phone, a tablet, smart glasses, a smart helmet, a smart watch, etc.), but it is understood that the device may also be a machine capable of autonomous movement, e.g., a drone, an unmanned automobile, a robot, etc. The device may have an image capture device (e.g., a camera) and/or a display medium (e.g., a display screen) mounted thereon.

In one embodiment of the invention, the invention may be implemented in the form of a computer program. The computer program may be stored in various storage media (e.g., hard disk, optical disk, flash memory, etc.), which when executed by a processor, can be used to implement the methods of the present invention.

In another embodiment of the invention, the invention may be implemented in the form of an electronic device. The electronic device comprises a processor and a memory in which a computer program is stored which, when being executed by the processor, can be used for carrying out the method of the invention.

References herein to "various embodiments," "some embodiments," "one embodiment," or "an embodiment," etc., indicate that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment," or the like, in various places throughout this document are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. Thus, a particular feature, structure, or characteristic illustrated or described in connection with one embodiment may be combined, in whole or in part, with a feature, structure, or characteristic of one or more other embodiments without limitation, as long as the combination is not logically inconsistent or workable. Expressions appearing herein similar to "according to a", "based on a", "by a" or "using a" mean non-exclusive, i.e. "according to a" may encompass "according to a only", as well as "according to a and B", unless specifically stated or clear from context that the meaning is "according to a only". In the present application, for clarity of explanation, some illustrative operational steps are described in a certain order, but one skilled in the art will appreciate that each of these operational steps is not essential and some of them may be omitted or replaced by others. It is also not necessary that these operations be performed sequentially in the manner shown, but rather that some of these operations be performed in a different order, or in parallel, as desired, provided that the new implementation is not logically or operationally unfeasible.

Having thus described several aspects of at least one embodiment of this invention, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be within the spirit and scope of the invention. Although the present invention has been described by way of preferred embodiments, the present invention is not limited to the embodiments described herein, and various changes and modifications may be made without departing from the scope of the present invention.

Claims

1. An interaction method based on an optical communication device comprises the following steps:

a server receives request information from a first device and information related to a location of the first device;

the server obtains the position information of the first equipment relative to the optical communication device through the information related to the position of the first equipment;

the server sets a virtual object having spatial position information associated with the first device, wherein the spatial position information of the virtual object is determined based on the position information of the first device relative to an optical communication apparatus;

the server provides the request information to the second device;

the server receives a response of the second device to the request information; and

the server sends information about the virtual object to a second device, including spatial position information of the virtual object, wherein the information about the virtual object is usable by the second device to render the virtual object on its display medium based on its position information and pose information relative to the optical communication apparatus.

2. The method of claim 1, wherein the optical communication device associated with the location information of the first device and the optical communication device associated with the location information of the second device are the same optical communication device.

3. The method of claim 1, wherein the optical communication device associated with the location information of the first device and the optical communication device associated with the location information of the second device are different optical communication devices, and wherein the different optical communication devices have a fixed relative positional relationship.

4. The method of any of claims 1-3, wherein obtaining the location information of the first device relative to the optical communication apparatus from the information related to the location of the first device comprises at least one of:

extracting position information of the first device relative to an optical communication apparatus from information relating to a position of the first device;

obtaining location information of the first device relative to an optical communication apparatus by analyzing information related to a location of the first device;

obtaining location information of the first device relative to an optical communication apparatus by querying using information about a location of the first device; or

The position information of the first device is obtained through the information related to the position of the first device, and the position information of the first device relative to the optical communication device is obtained through the position information and the position information of the optical communication device.

5. The method according to any of claims 1-3, wherein the information related to the position of the first device comprises position information of the first device relative to an optical communication apparatus, wherein the first device determines its position information relative to the optical communication apparatus by capturing an image comprising the optical communication apparatus using an image capturing device and analyzing the image.

6. The method of any one of claims 1-3,

the position information of the second device relative to the optical communication apparatus is obtained by: the second apparatus determines its position information relative to the optical communication device by capturing an image including the optical communication device using an image capturing device and analyzing the image; or

Attitude information of the second device relative to the optical communication apparatus is obtained by: determining pose information of the second device relative to the optical communication apparatus based on imaging of the optical communication apparatus on a display medium of the second device.

7. The method of any of claims 1-3, further comprising:

the server receiving new information from a first device regarding the location of the first device;

the server updating the position information of the first device relative to the optical communication apparatus based on the new information related to the position of the first device;

the server updates the spatial position information of the virtual object based on the updated position information of the first equipment relative to the optical communication device; and

the server sends the updated spatial position information of the virtual object to the second device so that the second device can render or update the virtual object on its display medium based on its position information and attitude information relative to the optical communication apparatus and the updated spatial position information of the virtual object.

8. The method of any of claims 1-3, further comprising:

the server receiving information from the second device relating to the location of the second device and determining location information of the second device relative to the optical communication apparatus;

the server sets another virtual object having spatial position information associated with the second device, wherein the spatial position information of the other virtual object is determined based on the position information of the second device relative to the optical communication apparatus; and

the server sends information related to the other virtual object to the first device, the information including spatial position information of the other virtual object, wherein the information related to the other virtual object is usable by the first device to render the other virtual object on its display medium based on its position information and posture information relative to the optical communication apparatus.

9. The method of any of claims 1-3, wherein the request message has an associated validity time range and/or a validity geographical range.

10. An interactive system based on optical communication devices, comprising:

one or more optical communication devices; and

a server configured to implement the method of any one of claims 1-9.

11. An interaction method based on an optical communication device comprises the following steps:

the second device receives request information from the first device from the server;

the second device sends a response to the request information to the server;

the second device receiving information related to a virtual object from the server, the information including spatial location information of the virtual object, the spatial location information being determined based on location information of the first device relative to an optical communication apparatus;

the second device determining its position information and attitude information relative to the optical communication apparatus; and

the second device renders the virtual object on its display medium based on its position information and pose information relative to the optical communication apparatus and the information related to the virtual object.

12. The method of claim 11, wherein the optical communication device associated with the location information of the second device and the optical communication device associated with the location information of the first device are the same optical communication device.

13. The method of claim 11, wherein the optical communication device associated with the location information of the second device and the optical communication device associated with the location information of the first device are different optical communication devices, and wherein the different optical communication devices have a fixed relative positional relationship.

14. The method of any one of claims 11-13,

15. The method according to any one of claims 11-13, further comprising:

the second device receiving updated spatial position information of the virtual object from the server; and

the second device renders or updates the virtual object on its display medium based on its position information and pose information relative to the optical communication apparatus and the updated spatial position information of the virtual object.

16. An apparatus for interaction based on optical communication means, configured to implement the method of any of claims 11-15.

17. A storage medium in which a computer program is stored which, when being executed by a processor, is operative to carry out the method of any one of claims 1-9 and 11-15.

18. An electronic device comprising a processor and a memory, the memory having stored therein a computer program operable, when executed by the processor, to carry out the method of any of claims 1-9 and 11-15.