TW202201224A - Interaction method based on optical communictation device, electric apparatus, and computer readable storage medium - Google Patents

Abstract

The present invention an interaction method based on optical communication device comprising: a server receives information about a position of a first device from the first device; the server obtains the position information of the first device via the information which is related with the position of a first device; the server disposed a virtual object include a spatial position information, and the virtual object correlates with the first device, wherein, the spatial position information of the virtual object confirmed for based on the position information of the first device; the server transmits the information about the virtual object to a second device, the information comprises the spatial position information of the virtual object, wherein the information about the virtual object shows on the display of the second device based on the position information and the posture information via the optical communication device.

Description

Interaction method based on optical communication device, electronic device, and computer-readable recording medium

The invention belongs to the field of information interaction, and in particular relates to an interaction method based on an optical communication device, an electronic device and a computer-readable recording medium.

The statements in this section are merely to provide background information related to the present invention for an understanding of the present invention and may not constitute prior art.

With the widespread popularity of the Internet (Internet), various industries are trying to use the Internet platform to develop new service delivery methods, and "Internet +" has become a research hotspot. A widely used service method is location-based services. In many application scenarios, one of the users or service providers must know the exact location of the other party in order to perform convenient service interaction. However, the prior art cannot solve this problem well.

Taking the instant express service as an example, after a user orders food from a certain restaurant through the Internet at a certain location outdoors, he can wait for the food delivery person to deliver the food to that location. However, the location provided by the user is usually an imprecise location (for example, a certain intersection, a certain street, a certain park, etc.), even if it is an accurate location, it may be difficult for the delivery person to determine due to the large number of people nearby Whoever is around is the user who ordered the meal, so it is necessary to conduct repeated telephone communication with the user.

Another example of location-based services is restaurant dining. After the user orders the meal, the corresponding table number and order information will be recorded by the restaurant. After the dishes are ready, the clerk in charge of delivering the food will deliver the dishes to the table with the corresponding table number. To this end, the clerk needs to keep in mind the location of each table number so that the food can be delivered to the table accurately and quickly. However, this method of relying only on the memory of the clerk is prone to errors in many cases (especially in the case of a large restaurant), resulting in the misdelivery of dishes, or the need for the clerk to confirm with the user when the dishes are delivered.

In order to solve the above problems, the present application proposes an interaction method and electronic device based on an optical communication device.

One aspect of the present invention relates to an interaction method based on an optical communication device, comprising: a server receiving information related to the location of the first device from a first device; information about obtaining location information of the first device; the server sets a virtual object with spatial location information associated with the first device, wherein the spatial location information of the virtual object is based on the first device The location information of the device is determined; the server sends the information related to the virtual object to the second device, the information includes the spatial location information of the virtual object, wherein the information related to the virtual object can is used by the second device to present the virtual object on the display medium of the second device based on the position information and attitude information determined by the second device through the optical communication device.

Further, obtaining the location information of the first device through the information related to the location of the first device includes at least one of the following: extracting the location information of the first device from the information related to the location of the first device. location information; the location information of the first device is obtained by analyzing the information related to the location of the first device; or, the first device is obtained by query using the information related to the location of the first device location information.

Further, the information related to the position of the first device includes position information of the first device relative to the optical communication device, wherein the first device captures the information including the optical communication device by using an image capturing device. image and analyze the image to determine position information of the first device relative to the optical communication device.

Further, the second device determines the position information and/or attitude information of the second device by capturing an image including the optical communication device using an image capturing device and analyzing the image.

Further, the method further includes: before the server sends the information related to the virtual object to the second device, the server determines the one or more related information that needs to be presented on the display medium of the second device. One or more virtual objects associated with the first device.

Further, the method further includes: the server receiving new information related to the location of the first device from the first device; the server is based on the new location of the first device The relevant information updates the location information of the first device; the server updates the spatial location information of the virtual object based on the updated location information of the first device; and the server updates all the updated location information. The spatial position information of the virtual object is sent to the second device, so that the second device can use the position information and attitude information of the second device and the updated spatial position information of the virtual object in the second device. The virtual object is rendered or updated on the device display medium.

Further, the method further includes: the server receives information related to the location of the second device from the second device, and determines the location information of the second device; the server is configured with another virtual object having spatial location information associated with the second device, wherein the spatial location information of the other virtual object is determined based on the location information of the second device; and the server communicates with the other virtual object Information about a virtual object is sent to the first device, the information includes spatial position information of the other virtual object, wherein the information about the other virtual object can be used by the first device to base on the The position information and posture information of the first device present the other virtual object on the display medium of the first device.

Further, before the server sends the information related to the virtual object to the second device, the method further includes: the server provides the information from the first device or a part of the information to the second device a second device; and the server receives a response from the second device to the information or a portion of the information from the first device.

Further, the method further includes: the server obtains attribute information of the first device; and the server sets the virtual device associated with the first device based on the attribute information of the first device The object, wherein the attribute information of the first device includes information related to the first device, information related to the user of the first device, and/or user-customized information of the first device Information.

Further, the position information of the first device is the position information relative to the optical communication device, the position information in the location coordinate system, or the position information in the world coordinate system; and/or, the second device's position information The position information and attitude information are position information and attitude information with respect to the optical communication device, position information and attitude information in a site coordinate system, or position information and attitude information in a world coordinate system.

Further, the optical communication device associated with the location information of the first device and the optical communication device associated with the location information of the second device are the same optical communication device or different optical communication devices, wherein all The different optical communication devices have a determined relative pose relationship.

Further, the posture of the virtual object can be adjusted according to changes in the position and/or posture of the second device relative to the virtual object.

Another aspect of the present invention relates to a computer-readable recording medium having stored therein a computer program that, when executed by a processor, can be used to implement the above-described method.

Another aspect of the present invention relates to an electronic device comprising a processor and a memory, wherein the memory stores a computer program, which can be used to implement the above method when the computer program is executed by the processor.

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Optical communication devices are also referred to as optical tags, and the two terms are used interchangeably herein. Optical tags can transmit information by emitting different lights, which have the advantages of long recognition distance and loose requirements for visible light conditions, and the information transmitted by optical tags can change with time, thus providing large information capacity and flexible configuration capabilities.

The optical label usually includes a controller and at least one light source, and the controller can drive the light source through different driving modes to transmit different information to the outside. FIG. 1 shows an exemplary optical label 100, which includes three light sources (a first light source 101, a second light source 102, and a third light source 103, respectively). The light tag 100 also includes a controller (not shown in FIG. 1 ) for selecting a corresponding driving mode for each light source according to the information to be communicated. For example, in different driving modes, the controller can use different driving signals to control the light-emitting manner of the light source, so that when the light tag 100 is photographed by a device with an imaging function, the image of the light source can present different images. Appearance (eg, different colors, patterns, brightness, etc.). By analyzing the imaging of the light sources in the optical tag 100 , the driving mode of each light source at the moment can be analyzed, so as to analyze the information transmitted by the optical tag 100 at the moment.

In order to provide corresponding services to users based on optical tags, each optical tag may be assigned an identification information (ID), which is used to uniquely identify or identify the optical tag by the manufacturer, manager or user of the optical tag, etc. Label. Usually, the light source can be driven by the controller in the optical tag to transmit the identification information, and the user can use the device to capture the image of the optical tag to obtain the identification information transmitted by the optical tag, so as to access the identification information based on the identification information. Corresponding services, for example, visit a web page associated with the identification information of the optical tag, obtain other information associated with the identification information (eg, location information of the optical tag corresponding to the identification information), and so on. The device can acquire an image containing the light label by capturing an image of the light label through the camera on it, and analyze the imaging of the light label (or each light source in the light label) in each image through the built-in application In order to identify the information transmitted by the optical label.

The optical tag can be installed in a fixed location and can store the identification information (ID) of the optical tag and any other information (eg location information) in the server. In reality, a large number of optical tags can be constructed into an optical tag network. FIG. 2 illustrates an exemplary optical label network including a plurality of optical labels and at least one server, wherein information related to each optical label may be stored on the server. For example, identification information (ID) or any other information for each light tag, such as service information related to the light tag, description information or attributes related to the light tag, such as the position information of the light tag, may be stored on the server , physical size information, physical shape information, attitude or orientation information, etc. The device can use the identification information of the identified optical tag to query the server for other information related to the optical tag. The location information of the optical tag may refer to the actual location of the optical tag in the physical world, which may be indicated by geographic coordinate information. A server can be a software program running on a computing device, a computing device, or a cluster of computing devices. Optical tags can be offline, ie, optical tags do not need to communicate with a server. Of course, it is understood that online optical tags capable of communicating with the server are also feasible.

Figure 3 shows a light tag placed over a restaurant door. When the user scans the optical tag with the device, the identification information conveyed by the optical tag can be recognized, and the identification information can be used to access corresponding services, for example, to visit the restaurant's web page associated with the identification information of the optical tag. Optical tags can be deployed in various places as needed, for example, on plazas, on store fronts, in restaurants.

The superposition of the virtual object to the real scene can be realized by using the light tag as an anchor point, for example, the virtual object can be used to accurately identify the location of the user or the device in the real scene. The virtual object may be, for example, an illustration, a picture, a text, a digit, an emoticon, a virtual three-dimensional object, a three-dimensional scene model, an animation, a video, and the like.

Taking the instant express service as an example to illustrate, when a user with a device (such as a mobile phone, smart glasses, etc.) is walking on a commercial street, he may want to buy a cup of coffee and wait for the coffee shop clerk to deliver the coffee to his location. The user can use his device to scan and identify the optical tags arranged on the facade of a certain cafe around him, and access the corresponding service to purchase a cup of coffee through the identified optical tag identification information. When a user scans an optical tag with his device, an image of the optical tag can be taken and analyzed for relative positioning to determine the position of the user (more precisely, the user's device) relative to the optical tag information, the relative location information may be sent to the cafe's server along with the coffee purchase request. After receiving the coffee purchase request from the user, the server of the coffee shop may set a virtual object, for example, the virtual object may be the order number "123" corresponding to the coffee purchase request. The server can also determine the spatial position information of the virtual object according to the received position information of the user equipment relative to the optical tag, for example, the position of the virtual object can be set to the position of the user equipment or the position of the user equipment. 1 meter above. After the coffee shop prepares the user's coffee, the clerk of the coffee shop can deliver the coffee. Figure 4 shows a schematic diagram of a store clerk delivering coffee to a user. During the delivery process, the clerk can scan the optical tag with their device (eg, mobile phone, smart glasses, etc.) to determine the location information and attitude information of the clerk's device relative to the optical tag. The server may send information about the virtual object, including its spatial location, to the clerk's device when the clerk is using the device to scan the optical label or at other times. In this way, the positional relationship of the virtual object relative to the clerk's device can be determined with the light tag as an intermediate anchor, and further based on the attitude of the clerk's device, the virtual object can be presented at an appropriate position on the display medium of the clerk's device (eg, a digital sequence" 123”). For example, the digital sequence "123" can be superimposed at a suitable position in the real scene displayed on the display screen of the clerk's equipment, and the position where the digital sequence "123" is located or about 1 meter below it is where the coffee buyer is. Location. FIG. 5 shows a schematic diagram of superimposing virtual objects on the display medium of the clerk's equipment. In this way, the optical tag can be used as an anchor to realize the accurate superposition of virtual objects in the real scene, thereby helping the cafe clerk to quickly find the location of the coffee buyer, so as to realize the coffee delivery. In one embodiment, coffee shop clerks can use smart glasses instead of cell phones during delivery for more convenient delivery.

Let’s take the restaurant delivery service as an example to illustrate. When a user with a device is dining in a restaurant, he can use his device to scan and identify a certain optical label arranged in the restaurant, and access it through the identified optical label identification information. corresponding meal service. When a user scans an optical tag with his device, an image of the optical tag can be taken and analyzed for relative positioning to determine the position of the user (more precisely, the user's device) relative to the optical tag information, the relative location information may be sent to the restaurant's server along with the order request. After receiving the ordering request from the user, the server of the restaurant may set a virtual object, for example, the virtual object may be the order number "456" corresponding to the ordering request. The server can also determine the spatial position information of the virtual object according to the received position information of the user equipment relative to the optical tag, for example, the position of the virtual object can be set to the position of the user equipment or the position of the user equipment. 1 meter above. When the restaurant prepares the user's dishes, the restaurant clerk can deliver the dishes. During the delivery process, restaurant clerks can use their devices (eg, mobile phones, smart glasses, etc.) to scan the optical tags to determine the location information and attitude information of the clerk's devices relative to the optical tags. The server may send information about the virtual object, including its spatial location, to the clerk's device when the clerk is using the device to scan the optical label or at other times. In this way, the positional relationship of the virtual object relative to the clerk's device can be determined with the light tag as an intermediate anchor, and further based on the attitude of the clerk's device, the virtual object can be presented at an appropriate position on the display medium of the clerk's device (eg, a digital sequence" 456”). For example, the digital sequence "456" can be superimposed at a suitable position in the real scene displayed on the display screen of the clerk's equipment, and the position where the digital sequence "456" is located or about 1 meter below it is the dish that needs to be delivered. The user's location. In this way, the light tag can be used as an anchor to realize the accurate superposition of virtual objects in the real scene, thereby helping the restaurant staff to quickly find the location of the orderer.

In one embodiment, the determined location information of the device may also be the location information of the device in other physical coordinate systems, and the physical coordinate system may be, for example, a site coordinate system (for example, for a certain room, building, park, etc. established coordinate system) or the world coordinate system. In this case, the optical tag may have position information and attitude information in the physical coordinate system, which may be calibrated and stored in advance. Through the position information of the device relative to the optical tag and the position information and attitude information of the optical tag itself in a certain physical coordinate system, the position information of the device in the physical coordinate system can be determined. The device can recognize the information (such as identification information) transmitted by the optical tag, and use the information to obtain (for example, by querying) the position information and attitude information of the optical tag in a certain physical coordinate system.

In one embodiment, the user may also order not by scanning and recognizing the optical tag, but may order in any other way. The user can also not determine the location information of the light tag by taking the image of the light tag, but can inform the restaurant server of its location by scanning the QR code on the table or directly sending the table number to the restaurant server, and the restaurant server The device can pre-store the location information of each dining table, and determine the user's location information based on the identification information of the QR code scanned by the user or the table number sent by the user.

Fig. 6 shows the interactive method based on the light tag according to one embodiment, the method includes the following steps:

Step 601: The server receives information from the first device, where the information includes location information of the first device.

The information from the first device may be product purchase information sent to the server by the user of the first device, but may also be any other information. The location information of the first device may be its location information relative to the optical label, or may be its location information in a certain physical coordinate system. When the first device sends its position information relative to the optical tag to the server, it can also send the identification information of the optical tag identified by scanning the optical tag.

The device may determine its position information relative to the optical tag in various ways, and the relative position information may include distance information and orientation information of the device relative to the optical tag. In one embodiment, the device may determine its positional information relative to the optical tag by acquiring an image including the optical tag and analyzing the image. For example, the device can determine the relative distance between the optical label and the device through the imaging size of the optical label in the image and optional other information (for example, the actual physical size information of the optical label, the focal length of the device's camera) (the larger the image, the greater the The closer the distance; the smaller the image, the further the distance). The device may obtain the actual physical size information of the optical tag from the server using the identification information of the optical tag, or the optical tag may have a uniform physical size and store the physical size on the device. The device may determine the orientation information of the device relative to the optical label by including perspective distortion of the optical label imaging in the image of the optical label and optionally other information (eg, the imaged location of the optical label). The device may obtain the physical shape information of the optical tag from the server using the identification information of the optical tag, or the optical tag may have a uniform physical shape and store the physical shape on the device. In one embodiment, the device can also directly obtain the relative distance between the optical tag and the device through a depth camera or a binocular camera installed on the device. The device can also use any other existing positioning method to determine its position information relative to the optical tag.

Step 602 : The server sets a virtual object with spatial location information associated with the first device, wherein the spatial location information of the virtual object is determined based on the location information of the first device.

After receiving information (eg, product purchase information) from the first device, the server may set up a virtual object associated with the first device. The virtual object may be, for example, the order number corresponding to the product purchase information sent by the first device, the name of the user who purchased the product, the identification information of the product to be delivered, a simple virtual icon, and the like. The spatial position information of the virtual object is determined according to the position information of the first device, and the spatial position information may be position information relative to the optical label, or may be position information in other physical coordinate systems. The spatial position of the virtual object may simply be determined as the position of the first device, or the spatial position of the virtual object may be determined as other positions, for example, other positions located near the position of the first device.

Step 603: The server sends information related to the virtual object to the second device, and the information includes spatial position information of the virtual object.

The information related to the virtual object is used to describe the relevant information of the virtual object, for example, it may include pictures, text, figures, icons, etc. Attitude information, etc. Based on this information, the device can present the corresponding virtual object. The information related to the virtual object includes spatial position information of the virtual object, which may be position information relative to the optical label (eg, distance information relative to the optical label and direction information relative to the optical label of the virtual object). In one embodiment, the information related to the virtual object may further include superimposed posture information of the virtual object, and the superimposed posture information may be the posture information of the virtual object relative to the light label, or its posture in the real world coordinate system Information.

In one embodiment, the server may directly transmit information related to the virtual object to the second device, eg, over a wireless link. In another embodiment, the second device may recognize the identification information transmitted by the optical tag when scanning the optical tag, and use the identification information of the optical tag to obtain information related to the virtual object from the server.

Step 604 : The second device presents the virtual object on the display medium of the second device based on the position information and attitude information determined by the optical tag and the information related to the virtual object.

The position information of the second device may be its position information relative to the optical label, or may be its position information in a certain physical coordinate system. The second device may use various methods to determine its position information relative to the optical tag, such as the various methods described above in step 601, which will not be repeated here.

The second device can determine its gesture information, and the gesture information can be used to determine the range or boundary of the real scene captured by the device. The attitude information of the second device may be its attitude information relative to the optical tag, or its attitude information in a certain physical coordinate system. Through the attitude information of the device relative to the optical tag and the position information and attitude information of the optical tag itself in a certain physical coordinate system, the attitude information of the device in the physical coordinate system can be determined. Usually, the posture information of the device is actually the posture information of the image acquisition device (such as a camera) of the device. In one embodiment, the second device can scan the optical tag, and can determine its attitude information relative to the optical tag according to the imaging of the optical tag, when the imaging position or imaging area of the optical tag is located in the center of the imaging field of view of the second device , it can be considered that the second device is currently facing the light tag. The direction of imaging of the optical tag can be further considered when determining the pose of the device. As the posture of the second device changes, the imaging position and/or imaging direction of the optical tag on the second device will change accordingly. Therefore, the relative position of the second device can be obtained according to the imaging of the optical tag on the second device. Attitude information on the light tag.

In one embodiment, the position information and attitude information (which may be collectively referred to as pose information) of the device relative to the optical tag can also be determined in the following manner. Specifically, a coordinate system may be established according to the optical label, and the coordinate system may be referred to as an optical label coordinate system. Some points on the optical label can be determined as some spatial points in the optical label coordinate system, and the coordinates of these spatial points in the optical label coordinate system can be determined according to the physical size information and/or physical shape information of the optical label. Some points on the light label may be, for example, the corners of the housing of the light label, the end of the light source in the light label, some identification points in the light label, and so on. According to the object structure feature or geometric structure feature of the optical tag, the image points corresponding to these spatial points can be found in the image captured by the device camera, and the position of each image point in the image can be determined. According to the coordinates of each spatial point in the optical tag coordinate system and the position of each corresponding image point in the image, combined with the internal reference information of the device camera, the position of the device camera in the optical tag coordinate system when the image is captured can be calculated. Attitude information (R, t), where R is the rotation matrix, which can be used to represent the attitude information of the device camera in the light tag coordinate system, and t is the displacement vector, which can be used to represent the device camera in the light tag coordinate system. location information. The method for calculating R and t is known in the prior art. For example, the 3D-2D PnP (Perspective-n-Point) method can be used to calculate R and t. In order not to obscure the present invention, it will not be described in detail here. . The rotation matrix R and the displacement vector t can actually describe how to convert the coordinates of a point between the light tag coordinate system and the device camera coordinate system. For example, through the rotation matrix R and the displacement vector t, the coordinates of a point in the light label coordinate system can be converted into the coordinates in the device camera coordinate system, and can be further converted into the position of the image point in the image. In this way, for a virtual object with multiple feature points (multiple points on the outline of the virtual object), the spatial position information of the virtual object may include the coordinates of the multiple feature points in the optical label coordinate system (that is, , relative to the position information of the optical tag), based on the coordinates of multiple feature points in the optical tag coordinate system, the coordinates of these feature points in the device camera coordinate system can be determined, so that the respective imaging of these feature points on the device can be determined. Location. Once the respective imaging positions of the multiple feature points of the virtual object are determined, the imaging position, size, or posture of the virtual object as a whole can be determined accordingly.

In one embodiment, the virtual object to be superimposed may have a preset imaging size. In one embodiment, when the superimposed information includes superimposed posture information of the virtual object, the posture of the superimposed virtual object may be further determined. In one embodiment, the imaging of the virtual object to be superimposed on the device can be determined according to the pose information (R, t) of the device (more precisely, the device's camera) relative to the optical label calculated above position, size, or attitude, etc. In one case, if it is determined that the virtual object to be superimposed is not currently in the field of view of the second device (eg, the imaging position of the virtual object is outside the display screen), the virtual object is not displayed.

In one embodiment, after the first device or the second device scans the optical tag, various sensors (eg, acceleration sensor, magnetic sensor, orientation sensor, gravity sensor) built into the device can be used, for example. (e.g., inertial navigation, visual odometry, SLAM, VSLAM, SFM, etc.) to measure or track its position change and/or attitude change by methods known in the art (eg, inertial navigation, visual odometry, SLAM, VSLAM, SFM, etc.) position and/or attitude.

In the above-mentioned embodiment, the light tag is actually used as an anchor point, and based on the anchor point, the accurate superposition of the virtual object in the real scene observed by the second device is realized. Devices can use a variety of feasible ways to render real-world scenarios. For example, the device can collect real-world information through a camera and use the above information to reproduce a real scene on a display screen, and images of virtual objects can be superimposed on the display screen. Devices (such as smart glasses) can also not reproduce the real scene through the display screen, but can simply reproduce the real scene through prisms, lenses, mirrors, transparent objects (such as glass), etc. The images of virtual objects can be optically superimposed on the real scene. The above-mentioned display screens, prisms, lenses, mirrors, transparent objects, etc. may be collectively referred to as the display medium of the device, and virtual objects may be presented on the display medium. For example, in an optical see-through augmented reality device, the user observes the real scene through a specific lens, and the lens can reflect the image of the virtual object into the user's eyes. In one embodiment, the user of the device can directly observe the real scene or parts thereof, which need not be reproduced through any medium before being observed by the user's eyes, and the virtual objects can be optically superimposed on the real scene. Thus, the actual scene or parts thereof do not necessarily need to be presented or reproduced by the device before being observed by the user's eyes.

After superimposing the virtual object, the device may be translated and/or rotated, in which case, methods known in the art can be used (eg, using the device's built-in accelerometer, gyroscope, visual odometry, etc. ) to track its position changes and attitude changes, so as to adjust the display of virtual objects. However, there may be errors in tracking position and attitude changes, so in one embodiment, the device can In some cases, the optical label is scanned at regular intervals to determine its position information and attitude information, and the imaging position and/or imaging size of the virtual object is re-determined, so as to correct the superposition of the virtual object in the real scene.

In one embodiment, after the virtual object is superimposed, the device or its user can perform operations on the virtual object to change the properties of the virtual object. For example, the device or its user may move the position of the virtual object, change the pose of the virtual object, change the size or color of the virtual object, add annotations to the virtual object, and so on. In one embodiment, after the device or its user changes the properties of the virtual object, the modified property information of the virtual object may be uploaded to the server. The server may update information about the virtual object stored in the server based on the modified attribute information. In one embodiment, the device or its user can perform a delete operation on the superimposed virtual object and notify the server. Still taking the coffee purchase application mentioned above as an example, after the coffee shop clerk carrying the second device completes the delivery of coffee to the user, the virtual digit sequence "123" associated with the user can be deleted.

In some embodiments, the information from the first device may not include the location information of the first device, and the server may obtain the location information of the first device in other ways. In one embodiment, the server may obtain location information of the first device by analyzing information from the first device. For example, the information from the first device may include an image captured by the first device containing the optical tag, and the server may obtain position information of the first device relative to the optical tag by analyzing the image. In one embodiment, the server may obtain location information of the first device by querying using information from the first device. For example, the information from the first device may be two-dimensional code identification information or identification information such as a table number, and based on the identification information, the server may obtain the location information of the first device by querying. Any information that can be used to obtain the location of the device (e.g., images taken by the device containing optical tags, QR code identification information scanned by the device, table numbers sent by the device, etc.) Information".

In one embodiment, multiple virtual objects may be presented simultaneously on the display medium of the second device. In one embodiment, the server may determine one or more virtual objects that need to be rendered on the display medium of the second device. For example, if the first clerk of the coffee shop wants to deliver coffee for the first user, the server can send the relevant information of the virtual object associated with the first user to the device of the first clerk; If the second clerk wants to deliver coffee to the second user and the third user at the same time, the server can send the relevant information of the virtual object associated with the second user and the relevant information of the virtual object associated with the third user to the second user. Shop assistant's equipment.

In some cases, the user may change his location after he has sent his location information to the server using the first device. For example, a user who purchased coffee may be walking around the neighborhood after sending their location along with the purchase request. In order to enable the server to know the latest location of the user or the first device in time, the new location information of the first device may be sent to the server. The first device can determine its latest location information in the various ways mentioned above (for example, by collecting an image including the light tag and analyzing the image), or it can use a built-in sensor in the first device (for example, acceleration) sensors, gyroscopes, etc.) to track changes in the position of the first device. New location information of the first device can be sent to the server periodically, or new location information can be activated when the difference between the new location of the first device and the last location sent to the server is greater than a predetermined threshold of sending. In this way, the server can know the new location information of the first device in time, and can update the spatial location information of the virtual object accordingly, and notify the second device of the new spatial location information of the virtual object. The second device may use the new spatial position information of the virtual object to render or update the virtual object on its display medium accordingly.

Fig. 7 shows an interaction method based on an optical tag according to an embodiment, the method can track the position of the first device, and steps 701-704 thereof are similar to steps 601-604 in Fig. 6, which are not repeated here . The interaction method of FIG. 7 further includes the following steps:

Step 705: The server receives new information from the first device.

The new information may be any information that can be used to obtain the position of the first device, for example, displacement information of the first device obtained by a sensor built in the first device through tracking.

Step 706: The server updates the location information of the first device based on the new information.

Step 707: The server updates the spatial location information of the virtual object based on the updated location information of the first device.

Step 708 : The server sends the updated spatial position information of the virtual object to the second device, so that the second device can display the updated spatial position information of the virtual object based on its position information and attitude information and the updated spatial position information of the virtual object. Render or update virtual objects on the medium.

In one embodiment, virtual objects associated with the second device may also be enabled to be rendered on the display medium of the first device. Taking the coffee purchase service described above as an example, during the delivery process, the clerk can scan the light tag with his device (eg, mobile phone, smart glasses, etc.) to determine the location information and attitude information of the clerk's device. After this, the clerk's device can send its location information to the server. The server may set a virtual object for the clerk's device, and the spatial location information of the virtual object is determined based on the location information of the clerk's device. The server can send information about the virtual object to the device of the user who purchased the coffee, and can notify the user that the coffee is being delivered. The user can then scan the light tag with his device (eg, cell phone, smart glasses, etc.) to determine the location information and attitude information of the user's device. Thus, the user equipment can present the virtual object (eg, the digit sequence "123") at the appropriate location on the display medium of the user equipment based on its position information and gesture information and information about the virtual object associated with the store clerk's equipment, This contributes to a more convenient interaction between the user and the clerk. Since the clerk delivering coffee is usually on the move, the location of the clerk's device can be tracked and the location of the clerk's device can be periodically or instantaneously sent to the server to update the spatial location information of the virtual object associated with the clerk's device, The updated spatial location information is then sent to the user's device.

FIG. 8 shows an interactive method based on a light tag according to an embodiment, the method can further present a virtual object associated with the second device on the display medium of the first device, and steps 801-804 thereof are the same as the steps of FIG. 6 . 601-604 are similar and will not be repeated here. The interaction method of FIG. 8 further includes the following steps:

Step 805: The server receives the information from the second device, and determines the location information of the second device.

Step 806 : The server sets another virtual object associated with the second device with spatial location information, wherein the spatial location information of the other virtual object is determined based on the location information of the second device.

Step 807: The server sends information related to the other virtual object to the first device, the information includes the spatial position information of the other virtual object, so that the first device can be based on its position information and attitude information and The information related to the other virtual object is presented on the display medium of the other virtual object.

In one embodiment, in the method shown in FIG. 8 , the location information of the second device and the spatial location information of the other virtual object may be further updated in a manner similar to the method of FIG. 7 , so that The other virtual object presented on the display medium of the first device can track the position of the second device.

In many scenarios, there may be more than one optical tag, but an optical tag network as shown in FIG. 2 , in which the server can know the pose information of each optical tag or the relative pose relationship between them. In these scenarios, the optical tags scanned by the first device and the second device may not be the same optical tag, and the first device may scan multiple different optical tags at different times to provide or update its position information (when providing or updating the optical tags). Updating the location information may send the identification information of the relevant optical tags), and the second device may also scan a plurality of different optical tags at different times to determine its location information and attitude information. For example, as shown in FIG. 9 , a plurality of optical labels including a first optical label and a second optical label may be arranged in a restaurant, and the user of the meal can scan the first optical label with the first device to determine its position, The restaurant clerk can use the second device to scan the second optical tag to determine the location information and attitude information of the second device when delivering the dish.

In some scenarios, the first device and the second device may be far apart initially. In this case, the user of the second device can first use some existing navigation methods (such as GPS navigation) to travel to the vicinity of the first device , and then, by using the second device to scan the surrounding light tags, the virtual object associated with the first device is presented on the display medium of the second device.

Figure 10 shows an application scenario that enables a location-based service solution between different individual users. The scene has an optical tag and a first user and a second user located near the optical tag, wherein the first user carries the first device, the second user carries the second device, the first device and the second user The device may be, for example, a mobile phone, smart glasses, and the like. The optical labels shown in FIG. 10 can be arranged on a square, for example. The first user on the plaza found himself temporarily in need of an item (eg, scissors, tape, medicine, etc.), but did not carry it with him. To this end, the first user may use the first device to scan and identify the optical tag to send a request to the server associated with the optical tag, eg, "Lend or ask for item A". When the first user scans the optical label using the first device, an image of the optical label can be captured, and relative positioning can be performed according to the image, so as to determine the first user (more precisely, the first user's first device), which may be sent to the server along with the above request. After receiving the request from the first device, the server may set a virtual object for the first device, and the virtual object may be, for example, an indicating arrow. The server can also determine the spatial position information of the virtual object according to the received position information of the first device, for example, the position of the virtual object can be set to the position of the first device or 1 meter above the position of the first device. . After the first device sends the request to the server, the second user can scan and identify the optical tag using the second device, and receive the request sent by the first device from the server associated with the optical tag. If the second user is able to provide item A to satisfy the first user's request, he can send a response to the server using the second device. After receiving the response from the second device, the server may send the relevant information (including its spatial position information) of the virtual object previously set for the first device to the second device. When the second device scans the optical label, its position information and attitude information can be determined. As such, virtual objects (eg, pointing arrows) may be presented at appropriate locations on the display medium of the second device based on the position and attitude of the second device. For example, an indication arrow can be superimposed at a suitable position in the real scene displayed on the display screen of the second device, and the position where the indication arrow is located or a position about 1 meter below it is where the first user or the first device is located. Location. FIG. 11 shows a schematic diagram of superimposing virtual objects (eg, pointing arrows) on the display medium of the second device. In this way, the light tag can be used as an anchor point to realize the accurate superposition of virtual objects, thereby helping the second user to quickly find the location of the first user, so as to realize the interaction between the two users. It can be understood that the optical labels scanned by the first user and the second user may be different optical labels.

FIG. 12 shows a light tag-based interaction method according to an embodiment, the method includes the following steps:

Step 1201: The server receives information from the first device, where the information includes location information of the first device.

The information from the first device may be, for example, a request for help sent by the user of the first device to the server, but may also be any other information. The information sent by the first device to the server may include identification information of the optical label.

Step 1202: The server sets a virtual object associated with the first device with spatial location information, wherein the spatial location information of the virtual object is determined based on the location information of the first device.

After receiving information (eg, a request for help) from the first device, the server may set up a virtual object associated with the first device. The spatial position information of the virtual object is determined according to the position information of the first device.

Step 1203: The server provides the information from the first device or a part thereof to the second device.

When the second device scans and recognizes the optical tag, it can interact with the server through the identification information of the recognized optical tag. During the interaction, the server may send the information it receives from the first device, or a portion thereof, to the second device. For example, the server may send "the first user asks for loan or purchase of item A" to the second device. Information from the first device may be presented to the second user on the second device in various forms, eg, in the form of a short message, in the form of a pop-up notification in an application, and the like. In one embodiment, a virtual message board may be presented on or near the light tag image presented on the display medium of the second device, and information from the first device may be displayed thereon.

Step 1204: The server receives a response from the second device to the information or a portion of the information from the first device.

If the second user is interested in information from the first device (eg, the second user can fulfill a request for help sent by the first user through the first device), he may send a response to the server through the second device.

Step 1205 : The server sends information related to the virtual object to the second device, where the information includes spatial position information of the virtual object.

After receiving the response from the second device, the server may send information related to the virtual object to the second device. The information related to the virtual object includes spatial position information of the virtual object, which may be position information relative to the light label. In one embodiment, the information related to the virtual object may further include superimposed gesture information of the virtual object.

Step 1206: The second device presents the virtual object on the display medium of the second device based on the position information and attitude information determined by the optical tag and the information related to the virtual object.

In some embodiments, the method shown in FIG. 12 may further track the location of the first device in a manner similar to the method shown in FIG. 7 . In some embodiments, the method shown in FIG. 12 may further present virtual objects associated with the second device on the display medium of the first device in a manner similar to the method shown in FIG. 8 .

In some embodiments, the information from the first device may have an associated valid time frame to define within which time frame the information is valid. For example, when the first user sends a request for help through the first device, the request may be set to be valid within a certain period of time (for example, 10 minutes). After the valid time frame is exceeded, the server may no longer send information from the first device, or parts thereof, to other devices.

In some embodiments, information from the first device may have an associated valid geographic range to define within which geographic range the information is valid. For example, in an application scenario, the first user sends an event notification to the server through the first device (for example, the first user is performing live at location A), so that other users can be invited to watch. The first user can set the geographic scope associated with the event notification. For example, the first user can set the event notification to be received by other users within a range of 500 meters, and the first user can also set the event notification to be received by other users interacting with light tags within a range of 500 meters. ,etc. In this way, the server can determine which users to send the event notification to based on the determined relative distances between other users and the first user, or the determined relative distances between different optical tags and the first user equipment.

In one embodiment, the server can obtain property information of the device, and set the virtual object associated with the device based on the property information of the device. The property information of the device may include information related to the device, information related to the user of the device information, and/or information customized by the user of the device, such as the device name or identification number; the device user's name, occupation, identity, gender, age, nickname, avatar, signature, account for an application on the device information, or information about operations performed by the user using the device, such as logging in to a webpage, registering an account, purchasing information, etc.

In one embodiment, where the virtual object has pose information, the pose of the virtual object can be adjusted with the position and/or attitude of the device relative to the virtual object, for example, so that a certain orientation of the virtual object (eg, the front of the virtual object) direction) always towards the device.

The device mentioned in this article can be a device carried or controlled by a user (eg, mobile phone, tablet, smart glasses, smart helmet, smart watch, etc.), but it is understood that the device can also be a machine that can move autonomously , for example, drones, driverless cars, robots, etc. An image capture device (such as a camera) and/or a display medium (such as a display screen) may be installed on the device.

In one embodiment of the present invention, the present invention may be implemented in the form of a computer program. The computer program can be stored in various computer-readable recording media (eg, hard disk, optical disk, flash memory, etc.), and when executed by the processor, can be used to implement the method of the present invention.

In another embodiment of the present invention, the present invention may be implemented in the form of an electronic device. The electronic device includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, the computer program can be used to implement the method of the present invention.

References herein to "various embodiments," "some embodiments," "one embodiment," or "an embodiment," etc. refer to the fact that a particular feature, structure, or property described in connection with the embodiment is included in the in at least one embodiment. Thus, appearances of the phrases "in various embodiments," "in some embodiments," "in one embodiment," or "in an embodiment" in various places throughout this document are not necessarily referring to the same Example. Furthermore, the particular features, structures, or properties may be combined in any suitable manner in one or more embodiments. Thus, particular features, structures, or properties shown or described in connection with one embodiment may be combined, in whole or in part, with the features, structures, or properties of one or more other embodiments without limitation, so long as the combination does not limit the Logical or not working. Expressions such as "according to A", "based on A", "by A" or "using A" appearing herein are meant to be non-exclusive, that is, "according to A" may encompass "according to A only" or "According to A and B" is covered unless specifically stated or clear from the context to mean "according to A only". In this application, for the sake of clarity, some schematic operation steps are described in a certain order, but those skilled in the art can understand that each of these operation steps is not essential, and some of them may be omitted or replaced by other steps. These operational steps do not necessarily have to be performed sequentially in the manner shown, rather, some of these operational steps may be performed in a different sequence according to actual needs, or concurrently, as long as the new execution manner is not illogical or inoperable.

Having thus described several aspects of at least one embodiment of this invention, it will be appreciated that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications and improvements are intended to be within the spirit and scope of the present invention. Although the present invention has been described in terms of the preferred embodiments, the present invention is not limited to the embodiments described herein, and various changes and changes can be made without departing from the scope of the present invention.

100: Light Label 101: The first light source 102: Second light source 103: The third light source 601-604: Steps 701-708: Steps 801-807: Steps 1201-1206: Steps

FIG. 1 is an exemplary optical label of the present invention.

FIG. 2 is an exemplary optical label network of the present invention.

Fig. 3 is an optical label arranged above a restaurant door according to the present invention.

FIG. 4 is a schematic diagram of a store clerk delivering coffee to a user according to the present invention.

FIG. 5 is a schematic diagram of superimposing a virtual object on the display medium of the shop assistant equipment according to the present invention.

FIG. 6 is an interactive method based on an optical tag according to an embodiment of the present invention.

FIG. 7 is an interactive method based on an optical tag according to an embodiment of the present invention.

FIG. 8 is an interaction method based on an optical tag according to an embodiment of the present invention.

FIG. 9 is an interactive system including two optical tags of the present invention.

FIG. 10 is an application scenario of the present invention, which can implement a location-based service solution among different individual users.

FIG. 11 is a schematic diagram of superimposing virtual objects on the display medium of the second device according to the present invention.

FIG. 12 is an interaction method based on an optical tag according to an embodiment of the present invention.

601-604: Steps

Claims (14)

An interaction method based on an optical communication device, comprising: a server receives information from a first device about the location of the first device; the server obtains the location information of the first device through information related to the location of the first device; The server sets a virtual object with spatial location information associated with the first device, wherein the spatial location information of the virtual object is determined based on the location information of the first device; The server sends information related to the virtual object to the second device, the information includes spatial location information of the virtual object, wherein the information related to the virtual object can be used by the second device The virtual object is presented on the display medium of the second device based on the position information and attitude information determined by the optical communication device based on the second device. The method of claim 1, wherein obtaining the location information of the first device through information related to the location of the first device includes at least one of the following: extracting location information of the first device from information related to the location of the first device; obtaining location information for the first device by analyzing information related to the location of the first device; or The location information of the first device is obtained by query using the information related to the location of the first device. The method of claim 1, wherein the information related to the position of the first device includes position information of the first device relative to the optical communication device, wherein the first device uses a graph The image capture device captures an image including the optical communication device and analyzes the image to determine position information of the first device relative to the optical communication device. According to the method described in item 1 of the claimed scope, wherein, The second device determines position information and/or attitude information of the second device by capturing an image including the optical communication device using an image capturing device and analyzing the image. The method as described in item 1 of the scope of the application, further comprising: Before the server sends the information about the virtual object to the second device, the server determines one or more information associated with the one or more first devices that need to be rendered on the display medium of the second device a virtual object. The method as described in item 1 of the scope of the application, further comprising: the server receives new information about the location of the first device from the first device; the server updates the location information of the first device based on the new information related to the location of the first device; the server updates the spatial location information of the virtual object based on the updated location information of the first device; and The server sends the updated spatial position information of the virtual object to the second device, so that the second device can be based on the position information and attitude information of the second device and the updated virtual object. The spatial position information of the object presents or updates the virtual object on the display medium of the second device. The method as described in item 1 of the scope of the application, further comprising: the server receives information about the location of the second device from the second device and determines the location information of the second device; the server sets another virtual object having spatial location information associated with the second device, wherein the spatial location information of the other virtual object is determined based on the location information of the second device; and The server sends information related to the other virtual object to the first device, the information includes spatial position information of the other virtual object, wherein the information related to the other virtual object can be The first device is used to present the other virtual object on the display medium of the first device based on the position information and attitude information of the first device. The method of claim 1, before the server sends the information related to the virtual object to the second device, further comprising: the server provides information from the first device, or a portion of the information, to the second device; and The server receives a response from the second device to information or a portion of the information from the first device. The method as described in item 1 of the scope of the application, further comprising: the server obtains attribute information of the first device; and The server sets the virtual object associated with the first device based on the attribute information of the first device, Wherein, the attribute information of the first device includes information related to the first device, information related to the user of the first device, and/or information customized by the user of the first device. The method as described in item 1 of the claimed scope, wherein, The location information of the first device is location information relative to the optical communication device, location information in a site coordinate system, or location information in a world coordinate system; and/or, The position information and attitude information of the second device are the position information and attitude information relative to the optical communication device, the position information and attitude information in the site coordinate system, or the position information and attitude information in the world coordinate system. The method of claim 10, wherein the optical communication device associated with the location information of the first device and the optical communication device associated with the location information of the second device are the same optical communication device or different optical communication devices, wherein the different optical communication devices have a determined relative pose relationship. The method of claim 1, wherein the posture of the virtual object can be adjusted according to changes in the position and/or posture of the second device relative to the virtual object. A computer-readable recording medium in which a computer program is stored, when the computer program is executed by a processor, can be used to implement the method described in any one of items 1 to 12 of the patent application scope. An electronic device, comprising a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the computer program can be used to realize the invention as described in any one of items 1 to 12 of the patent application scope. method described.

Images