WO2020192543A1

WO2020192543A1 - Method for presenting information related to optical communication apparatus, and electronic device

Info

Publication number: WO2020192543A1
Application number: PCT/CN2020/080160
Authority: WO
Inventors: 方俊; 牛旭恒; 王强; 李江亮
Original assignee: 北京外号信息技术有限公司
Priority date: 2019-03-27
Filing date: 2020-03-19
Publication date: 2020-10-01
Also published as: CN111753565B; TW202103045A; CN111753565A

Abstract

Provided are a method for presenting information related to an optical communication apparatus, and an electronic device. The method comprises: using a first camera to obtain, in an optical communication apparatus recognition mode, a first image containing an optical communication apparatus; obtaining, based on the first image, position information of the optical communication apparatus relative to the first camera; obtaining, according to the position information of the optical communication apparatus relative to the first camera, position information of the optical communication apparatus relative to a second camera; using the second camera to obtain a second image containing the optical communication apparatus; and presenting, according to the position information of the optical communication apparatus relative to the second camera and on the second image, information related to the optical communication apparatus.

Description

Method and electronic equipment for presenting information related to optical communication device

Technical field

The present invention belongs to the field of optical information technology, and in particular relates to a method and electronic equipment for presenting information related to an optical communication device.

Background technique

The statements in this section are only to provide background information related to the present invention to help understand the present invention, and this background information does not necessarily constitute prior art.

Optical communication devices are also called optical tags, and these two terms can be used interchangeably in this article. Optical tags can transmit information by emitting different lights. They have the advantages of long recognition distance, relaxed requirements for visible light conditions, and strong directivity, and the information transmitted by optical tags can change over time, which can provide large information capacity and flexibility Configuration capabilities. Compared with the traditional two-dimensional code, the optical tag has a longer recognition distance and stronger information interaction ability, which can provide huge convenience for users and businesses.

The optical label recognition device can be, for example, a device carried or controlled by a user (for example, a mobile phone with a camera, a tablet computer, smart glasses, a smart helmet, a smart watch, a car, etc.), or a machine that can move autonomously (for example, Drones, driverless cars, robots, etc.). In many cases, in order to identify the information transmitted by the optical tag or to avoid interference from ambient light, the identification device needs to use the camera on it to capture the image of the optical tag in a specific optical tag recognition mode (for example, low exposure mode). Obtain the image of the optical label, and analyze these images through the built-in application to identify the information conveyed by the optical label. However, an image containing a light tag captured in a specific light tag recognition mode usually cannot reproduce the environmental information around the light tag well, which is very unfavorable for user experience or subsequent user interaction. For example, for an image containing a light tag captured in a low exposure mode, the brightness of the image surrounding the light tag is usually very low, or even pitch black. When such an image is displayed on the display screen of the optical tag recognition device, it will affect the user's interactive experience. Figure 1 shows an exemplary image containing a light label taken in low exposure mode. The middle position of the upper part of the figure has the image of the light label. However, since the objects around the light label are usually low in brightness, It is difficult to distinguish objects around the light tag from the image taken in this low exposure mode. Although the image obtained in the normal shooting mode can show the environmental information around the light tag, because the image is not shot in the light tag recognition mode, the recognition of the light tag cannot be achieved based on the image (that is, the light cannot be recognized). The information transmitted by the label or the optical label), so the interactive information corresponding to the optical label (for example, an interactive icon) cannot be presented on the image.

Therefore, there is a need for an improved method and electronic device for presenting information related to optical tags.

Summary of the invention

One aspect of the present invention relates to a method for presenting information related to an optical communication device, including: using a first camera to obtain a first image containing an optical communication device in an optical communication device recognition mode; based on the first image Obtain the position information of the optical communication device relative to the first camera; obtain the position information of the optical communication device relative to the second camera according to the position information of the optical communication device relative to the first camera; use the second camera to obtain A second image of the optical communication device; and presenting information related to the optical communication device on the second image according to position information of the optical communication device relative to the second camera.

Optionally, wherein, obtaining the position information of the optical communication device relative to the second camera according to the position information of the optical communication device relative to the first camera includes: according to the position information of the optical communication device relative to the first camera , Using the rotation matrix and displacement vector between the first camera and the second camera to obtain the position information of the optical communication device relative to the second camera.

Optionally, wherein obtaining the position information of the optical communication device relative to the first camera based on the first image includes: obtaining the relative position of the optical communication device based on the imaging of the optical communication device in the first image Location information of the first camera.

Optionally, wherein the obtaining the position information of the optical communication device relative to the first camera based on the imaging of the optical communication device in the first image includes: based on the information of the optical communication device in the first image Obtain the distance of the optical communication device relative to the first camera based on the imaging size; obtain the direction of the optical communication device relative to the first camera based on the imaging position of the optical communication device in the first image; The distance and direction of a camera are used to obtain the position information of the optical communication device relative to the first camera.

Optionally, wherein the obtaining the position information of the optical communication device relative to the first camera based on the imaging of the optical communication device in the first image includes: according to some points on the optical communication device in the optical communication device The coordinates in the coordinate system and the imaging positions of these points in the first image are combined with the internal parameter information of the first camera to obtain the position information of the optical communication device relative to the first camera.

Optionally, the above method further includes: obtaining posture information of the optical communication device relative to the first camera based on the first image; obtaining the optical communication device according to posture information of the optical communication device relative to the first camera The posture information relative to the second camera, and wherein the presenting the information related to the optical communication device on the second image according to the position information of the optical communication device relative to the second camera includes: according to the The position information and posture information of the optical communication device relative to the second camera present information related to the optical communication device on the second image.

Optionally, wherein, obtaining the posture information of the optical communication device relative to the first camera based on the first image includes: obtaining the relative position of the optical communication device based on imaging of the optical communication device in the first image The posture information of the first camera.

Optionally, wherein the obtaining the posture information of the optical communication device relative to the first camera based on the imaging of the optical communication device in the first image includes: determining the optical communication device in the first image The perspective deformation of the imaging to obtain the posture information of the optical communication device relative to the first camera.

Optionally, wherein the obtaining the posture information of the optical communication device relative to the first camera based on the imaging of the optical communication device in the first image includes: according to some points on the optical communication device The coordinates in the coordinate system and the imaging positions of these points in the first image obtain posture information of the optical communication device relative to the first camera.

Optionally, wherein the coordinates of some points on the optical communication device in the optical communication device coordinate system are determined according to the physical size information and/or physical shape information of the optical communication device.

Optionally, wherein, presenting the information related to the optical communication device on the second image according to the position information of the optical communication device relative to the second camera includes: according to the position information of the optical communication device relative to the second camera Determine the imaging position in the second image corresponding to the position information, and present information related to the optical communication device at the imaging position in the second image.

Optionally, wherein the second image is a real scene image with normal exposure.

Optionally, the above method further includes: obtaining identification information of the optical communication device based on the first image.

Another aspect of the present invention relates to a method for presenting information related to an optical communication device, including: using a first camera to obtain a first image containing an optical communication device in an optical communication device recognition mode; and obtaining the optical communication device The first imaging position of the device in the first image; using a second camera to obtain a second image containing the optical communication device; and presenting at the second imaging position in the second image according to the first imaging position Information related to the optical communication device, wherein the first camera and the second camera are installed on the same plane and have the same posture and internal parameters.

Optionally, wherein the second imaging position is the same as the first imaging position.

Optionally, wherein, according to the relative offset between the first camera and the second camera, the first imaging position, the Z coordinate of the optical communication device in the first camera coordinate system or the second camera coordinate system, or The vertical distance between the optical communication device and the installation plane of the first camera and the second camera determines the second imaging position.

Optionally, wherein the second camera is determined according to the relative offset between the first camera and the second camera, the first imaging position, and the distance from the optical communication device to the first camera or the second camera. Imaging position.

Another aspect of the present invention relates to a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it can be used to implement the above method.

Another aspect of the present invention relates to an electronic device, which includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, it can be used to implement the above method.

The solution of the present invention provides a method for presenting information related to an optical communication device. By this method, an optical label can be displayed on an image taken in a non-optical label recognition mode (for example, a real image taken in a normal mode). Related information, therefore, enables users not only to interact with the optical tag through the image presented by the device, but also to perceive the environmental information around the optical tag, thereby improving the interaction efficiency and the interaction experience.

Description of the drawings

The following further describes the embodiments of the present invention with reference to the drawings, in which:

Figure 1 shows an exemplary image containing a light tag taken in a low exposure mode;

Figure 2 shows an exemplary optical label;

Figure 3 shows an image of a light label taken by a rolling shutter imaging device in a low exposure mode;

Figure 4 shows a method for presenting information related to optical tags according to an embodiment of the present invention;

Figure 5 shows an exemplary normally exposed image containing a light label;

Fig. 6 shows an exemplary image presented according to an embodiment of the present invention; and

Fig. 7 shows a method for presenting information related to an optical tag according to another embodiment of the present invention.

detailed description

In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the present invention in detail through specific embodiments with reference to the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, but not to limit the present invention.

The optical tag usually includes a controller and at least one light source, and the controller can drive the light source through different driving modes to transmit different information outward. In order to provide corresponding services to users and businesses based on light tags, each light tag can be assigned an identification information (ID), which is used to uniquely identify or identify the manufacturer, manager, or user of the light tag Light label. Generally, the controller in the optical tag can drive the light source to transmit the identification information outward, and the user can use the optical tag identification device to perform continuous image collection on the optical tag to obtain the identification information transmitted by the optical tag, which can be based on the identification. Information to access corresponding services, for example, accessing a web page associated with the identification information of the optical tag, obtaining other information associated with the identification information (for example, location information of the optical tag corresponding to the identification information), and so on.

Fig. 2 shows an exemplary optical label 100, which includes three light sources (respectively a first light source 101, a second light source 102, and a third light source 103). The optical label 100 also includes a controller (not shown in FIG. 2) for selecting a corresponding driving mode for each light source according to the information to be transmitted. For example, in different driving modes, the controller can use driving signals with different frequencies to control the turning on and off of the light source, so that when a rolling shutter imaging device (such as a CMOS imaging device) is used to photograph the light label 100 in a low exposure mode When, the image of the light source can show different stripes. FIG. 3 shows an image of the optical label 100 taken by the rolling shutter imaging device in the low exposure mode when the optical label 100 is transmitting information, in which the image of the first light source 101 shows relatively narrow stripes. The images of the second light source 102 and the third light source 103 exhibit relatively wide fringes. By analyzing the imaging of the light source in the optical label 100, the driving mode of each light source at the moment can be analyzed, so as to analyze the information transmitted by the optical label 100 at the moment.

The optical label may additionally include one or more positioning marks located near the light source for transmitting information. The positioning marks may be, for example, lights of a specific shape or color, and the lights may, for example, remain on during operation.

The optical label recognition device can be, for example, a device carried or controlled by a user (for example, a mobile phone with a camera, a tablet computer, smart glasses, a smart helmet, a smart watch, a car, etc.), or a machine that can move autonomously (for example, Drones, driverless cars, robots, etc.). The optical label recognition device can obtain multiple images containing the optical label through continuous image acquisition of the optical label through the camera on it, and analyze the imaging of the optical label (or each light source in the optical label) in each image. Identify the information transmitted by the optical tag.

The identification information (ID) of the optical tag and any other information can be stored in the server. The other information is, for example, service information related to the optical tag, description information or attribute information related to the optical tag, such as location information of the optical tag, Physical size information, physical shape information, orientation information, etc. The optical label may also have unified or default physical size information and physical shape information. The device can use the identified identification information of the optical label to query the server to obtain other information related to the optical label. The server may be a software program running on a computing device, a computing device, or a cluster composed of multiple computing devices. The optical tag may be offline, that is, the optical tag does not need to communicate with the server. Of course, it can be understood that online optical tags that can communicate with the server are also feasible.

Figure 4 shows a method for presenting information related to a light tag according to an embodiment of the present invention. The method can be executed by a device that uses two cameras (referred to as the first camera and the second camera, respectively) , Wherein the first camera is used to operate in the light tag recognition mode to recognize the light tag, and the second camera is used to capture an image containing the light tag (for example, a real-life image under normal exposure). The position and posture of the first camera and the second camera may have a fixed relative relationship. In one embodiment, the first camera and the second camera may be installed on the same device (for example, a mobile phone with at least two cameras). However, either or both of the first camera and the second camera may not be installed on the device, but may be communicatively connected with the device. The rotation matrix R0 and the displacement vector t0 between the first camera and the second camera (also referred to as the first camera coordinate system and the second camera coordinate system) and the internal parameter information of the two cameras can be determined in advance. The rotation matrix R0 is used to represent the relative posture information between the two cameras, and the displacement vector t0 is used to represent the relative displacement information between the two cameras. By using the rotation matrix R0 and the displacement vector t0, the position information in the first camera coordinate system can be converted into the position information in the second camera coordinate system through a rotation operation and a displacement operation. In some devices with two cameras (for example, a mobile phone with two cameras), the postures of the two cameras are the same. In this case, the rotation matrix R0 is the identity matrix, so in fact, two cameras are not required. Rotation operations between coordinate systems. The method includes the following steps:

Step 401: Use the first camera to obtain a first image containing the optical label in the optical label recognition mode.

When the first camera works in the optical tag recognition mode, it can take a first image containing the optical tag (for example, the image shown in FIG. 1). By analyzing the first image, the imaging position of the optical tag and the information transmitted by the optical tag can be obtained. The optical label recognition mode is usually different from the normal shooting mode of the camera. For example, in the optical label recognition mode, the camera can be set to a predetermined low exposure mode so that the information conveyed by the optical label can be recognized from the first image taken. After the information conveyed by the optical tag is recognized, interactive information (for example, an interactive icon) corresponding to the optical tag may be presented at the imaging position of the optical tag for the user to operate. However, the first image may not be user-friendly, because it may be difficult for the user to obtain useful other information, such as information about the surrounding environment of the light tag, when observing the first image with naked eyes. Therefore, directly presenting the interactive information corresponding to the light tag on the image obtained by using the light tag recognition mode cannot provide a good user experience.

In addition, it should be noted that although the optical label recognition mode is usually different from the normal shooting mode of the camera, the present invention does not exclude a solution in which the optical label recognition mode is the same or substantially the same as the normal shooting mode.

Step 402: Obtain position information of the optical label relative to the first camera based on the first image.

The position information of the optical label relative to the first camera can be obtained by analyzing the imaging of the optical label in the first image, which can be expressed as the position information of the optical label in the first camera coordinate system. For example, the position information may be represented by coordinates (X1, Y1, Z1) in a coordinate system with the first camera as the origin, and may be referred to as a displacement vector t1. The position information of the optical label can be represented by the position information of one point, for example, the position information of the optical label can be represented by the position information of the center point of the optical label; the position information of the optical label can also be represented by the position information of multiple points. Representation, for example, the position information of the optical label can be represented by the position information of multiple points that can define the rough outline of the optical label; the position information of the optical label can also be represented by the position information of a region; and so on.

In an embodiment, the distance and direction of the optical label relative to the first camera can be determined by analyzing the imaging of the optical label in the first image, thereby determining its position information relative to the first camera. For example, the relative distance between the optical label and the first camera can be determined by the imaging size of the optical label in the first image and optional other information (for example, the actual physical size information of the optical label, camera internal parameters) (the larger the image, the greater the distance The closer; the smaller the image, the farther the distance). The device where the first camera is located may obtain the actual physical size information of the optical label from the server, or the optical label may have a default uniform physical size (which may be stored on the device). The direction of the optical label relative to the first camera can be determined by analyzing the imaging position of the optical label in the first image.

Additionally or alternatively, the perspective deformation of the optical label imaging can be determined by analyzing the imaging of the optical label in the first image, thereby determining the posture information of the optical label relative to the first camera (also referred to as direction information). Or orientation information), for example, the posture information of the optical tag in the first camera coordinate system. The device where the first camera is located may obtain the actual physical shape information of the optical label from the server, or the optical label may have a default unified physical shape (which may be stored on the device). The determined posture information of the optical tag relative to the first camera can be represented by a rotation matrix R1. The rotation matrix is known in the imaging field, and in order not to obscure the present invention, it will not be described in detail here.

In an embodiment, a coordinate system can be established based on the optical tag, and the coordinate system can be called the world coordinate system or the optical tag coordinate system. Some points on the optical label can be determined as some spatial points in the world coordinate system, and the coordinates of these spatial points in the world coordinate system can be determined according to the physical size information and/or physical shape information of the optical label. The device where the first camera is located can obtain the physical size information and/or physical shape information of the optical tag from the server, or the optical tag can have default unified physical size information and/or physical shape information, and the device can store the physical size information And/or physical shape information. Some points on the optical label may be, for example, the corner of the housing of the optical label, the end of the light source in the optical label, some identification points in the optical label, and so on. According to the object structure feature or geometric structure feature of the optical tag, the image points corresponding to these spatial points can be found in the first image, and the position of each image point in the first image can be determined. According to the coordinates of each spatial point in the world coordinate system and the position of each corresponding image point in the first image, combined with the internal parameter information of the first camera, the position information of the optical tag in the first camera coordinate system can be calculated. It can be represented by the displacement vector t1. Additionally or alternatively, according to the coordinates of each spatial point in the world coordinate system and the position of each corresponding image point in the first image, the posture information of the optical tag in the first camera coordinate system can also be calculated. It can be represented by the rotation matrix R1. The combination (R1, t1) of the rotation matrix R1 and the displacement vector t1 is the pose information (that is, position and posture information) of the optical tag in the first camera coordinate system. The method of calculating the rotation matrix R and the displacement vector t according to the coordinates of each spatial point in the world coordinate system and the position of the corresponding image point in the image is known in the prior art. For example, 3D-2D can be used The PnP (Perspective-n-Point) method is used to calculate R and t. In order not to obscure the present invention, the detailed description is omitted here. The rotation matrix R and the displacement vector t can actually describe how to transform the coordinates of a certain point between the world coordinate system and the camera coordinate system. For example, through the rotation matrix R and the displacement vector t, the coordinates of a certain space point in the world coordinate system can be converted to the coordinates in the camera coordinate system, and can be further converted to the position of the image point in the image.

In an embodiment, the information conveyed by the optical label may be further obtained based on the imaging of the optical label in the first image, for example, identification information of the optical label.

Step 403: Obtain the position information of the optical label relative to the second camera according to the position information of the optical label relative to the first camera.

After obtaining the position information of the optical tag relative to the first camera (for example, the position information represented by the displacement vector t1), the relative pose information between the first camera and the second camera (for example, The rotation matrix R0 and the displacement vector t0 between the first camera and the second camera are obtained to obtain the position information of the optical label relative to the second camera. The position information may be, for example, the position information of the optical label in the second camera coordinate system. And can be represented by the displacement vector t2.

Additionally or alternatively, if the posture information of the optical tag relative to the first camera (for example, posture information represented by the rotation matrix R1) is also obtained in step 402, the first camera and the second camera can be used according to the posture information. The relative posture information between the cameras (for example, the rotation matrix R0 between the first camera and the second camera), to obtain the posture information of the optical label relative to the second camera, for example, the posture of the optical label in the second camera coordinate system Information, the posture information can be represented by the rotation matrix R2.

Step 404: Use the second camera to obtain a second image containing the light tag.

The second image containing the light tag obtained by the second camera may be, for example, a normal exposure real scene image, which contains information about the light tag and its surrounding environment. Figure 5 shows an exemplary normally exposed image containing a light label, which corresponds to the low exposure image shown in Figure 1, which shows a restaurant door, and a rectangular light label above the door . Although the second image can show the environmental information around the optical tag (that is, the second image is user-friendly), since the second image is not captured in the optical tag recognition mode, it cannot be based on the second image. The recognition of the optical label is realized (that is, the optical label or the information transmitted by the optical label cannot be identified), and thus the information corresponding to the optical label (for example, an interactive icon) cannot be presented on the second image.

The second image containing the light label obtained by the second camera is preferably a real-life image with normal exposure, but this is not a limitation. According to actual needs, the second image can also be an image obtained by the camera in other shooting modes, such as grayscale Images etc.

Step 405: According to the position information of the optical label relative to the second camera, present information related to the optical label on the second image.

In an embodiment, after obtaining the position information of the optical label relative to the second camera, the second image captured by the optical label on the second camera can be calculated using the imaging formula based on the position information and the internal parameter information of the second camera. The display position where the above should be. It is well known in the art to use an imaging formula to calculate the imaging position of a certain point based on its position information relative to the camera, and will not be described in detail here to avoid obscuring the present invention. According to the calculated display position, information related to the light tag may be presented (for example, superimposed, embedded, covered, etc.) at a suitable position on the second image, and the suitable position is preferably the calculated display position. The information related to the light tag can be various information, for example, the image of the light tag, the logo of the light tag, the identification information of the light tag, the icon associated with the light tag or its identification information, and the light tag or its identification information. The name of the associated store, any other information associated with the light label or its identification information, and various combinations of them.

Additionally or alternatively, if the posture information of the optical tag relative to the second camera is also obtained in step 403, when the information related to the optical tag is presented on the second image, the posture information may be further based on the posture information. To present information related to the light tag. For example, when the image, logo, icon, etc. of the light tag are presented, the posture information of these images, logos, icons, etc. can be set based on the posture information of the light tag in the second camera coordinate system. This is advantageous, especially when the logo or icon or the like corresponding to the light label is a three-dimensional virtual object.

Fig. 6 shows an exemplary image presented according to an embodiment of the present invention, which is superimposed on the image shown in Fig. 5 with a circular icon associated with the light label therein. The display position is the actual imaging position of the optical label. The icon can have an interactive function. After clicking the icon, the user can access the information of the corresponding restaurant, and can perform operations such as reservation, queuing, and ordering. In this way, the user can not only interact with the optical tag through the image presented by the device, but also perceive the environmental information around the optical tag.

It can be understood that the steps of obtaining the first image and obtaining the second image described above can be performed in any suitable order and can be performed concurrently. In addition, these steps can also be repeated as needed to continuously update the scene displayed by the camera and the display position of the light tag.

According to an embodiment of the present invention, the first camera and the second camera of the device are installed on the same plane (that is, the coordinate system of the first camera and the coordinate system of the second camera are not offset in the Z-axis direction), and have the same Internal parameters and the same posture (that is, the posture of the first camera coordinate system and the second camera coordinate system are the same, and the rotation matrix between the two is the identity matrix, so there is no need to perform the rotation operation between the two camera coordinate systems) For example, a mobile phone equipped with two cameras with the same internal parameters, the orientation of the two cameras is also the same, only the installation position is somewhat different (for example, the installation position of the two is offset by a few millimeters). In this case, the imaging position of the optical tag in the images captured by the two cameras is basically the same (especially when the optical tag is far from the camera), and there is only an insignificant offset. In view of this situation, in an embodiment of the present invention, the imaging position of the optical label in the image captured by the first camera (working in optical label recognition mode) can be directly used to determine the optical label in the second camera (working in non- Optical label recognition mode) the imaging position in the captured image. Fig. 7 shows a method for presenting information related to an optical tag according to this embodiment, which may include the following steps:

Step 701: Use the first camera to obtain a first image containing the optical label in the optical label recognition mode.

Step 702: Obtain the first imaging position of the optical label in the first image.

The imaging position of the light tag in the image can be represented by the position information of a point, for example, the imaging position of the light tag in the image can be represented by the position information of the center point of the light tag; the imaging position of the light tag in the image is also It can be represented by the position information of multiple points, for example, the imaging position of the light tag in the image can be represented by the position information of multiple points that can define the approximate outline of the light tag; the imaging position of the light tag in the image can also be It is represented by the location information of an area; etc.

Step 703: Use the second camera to obtain a second image containing the optical tag.

Step 704: Present information related to the light tag at a second imaging position in the second image according to the first imaging position.

In one embodiment, in step 704, the second imaging position may be derived based on the first imaging position according to the imaging formula.

Specifically, for two cameras with the same internal parameters and the same posture installed on the same plane, it is assumed that the two camera coordinate systems only have an offset d in the X-axis direction. Thus, the displacement vector T between the two camera coordinate systems It is (d, 0, 0). For a point (X, Y, Z) in the coordinate system of the first camera, its coordinates in the coordinate system of the second camera are (X+d, Y, Z). According to the imaging formula, the imaging position (u1, v1) of the point in the image taken by the first camera and the imaging position (u2, v2) in the image taken by the second camera can be calculated as:

=>

Combine the above formula to get:

Among them, fx, fy are the focal length of the camera in the x and y directions; (cx, cy) are the coordinates of the camera aperture in the camera coordinate system.

It can be seen from the above that for the same point in space, when the above two camera coordinate systems only have an offset d in the X-axis direction (for example, when the two cameras are arranged horizontally, the offset d can be preset Determination), for their imaging positions (u1, v1) and (u2, v2), exist as well.

Similarly, for the same point in space, when the above two camera coordinate systems only have an offset d in the Y-axis direction (for example, when the two cameras are arranged vertically, the offset d can be determined in advance) , For their imaging positions (u1, v1) and (u2, v2), there are as well.

Correspondingly, for the same point in space, if the above two camera coordinate systems have offsets dx and dy in the X-axis and Y-axis directions respectively (which can be determined in advance), their imaging positions (u1, v1 ) And (u2,v2), exist as well.

For the device with the first camera and the second camera installed, fx and fy are the parameter values inherent to the camera. Therefore, as long as the relative offset (including the offset direction and the offset distance) between the first camera and the second camera can be known in advance, after obtaining a certain point in the first camera coordinate system or the second camera coordinate system After the Z coordinate, the imaging position of the point in the image captured by another camera can be derived based on the imaging position of the point in the image captured by another camera. Since the first camera and the second camera are installed on the same plane (that is, there is no offset between the first camera coordinate system and the second camera coordinate system in the Z-axis direction), the point is in the first camera coordinate system or the second camera The Z coordinate in the coordinate system is the same, which is roughly equal to the vertical distance from the point to the installation plane of the camera. Therefore, in an embodiment, the Z coordinate of the optical label in the first camera coordinate system or the second camera coordinate system (or the vertical distance between the optical label and the installation plane of the two cameras) and the optical label in the first camera The first imaging position in the captured first image is used to determine the second imaging position of the light tag in the second image captured by the second camera. Any method described in step 402 above can be used to obtain the Z coordinate of the optical label in the camera coordinate system or the vertical distance from the optical label to the installation plane of the camera.

In some applications, when scanning and identifying the optical label, the optical label is usually located near the center of the screen. In this case, the distance between the optical label and the first camera or the second camera can be used as the distance between the optical label and the two cameras. The approximate value of the vertical distance of the installation plane. The distance from the optical label to the camera is easier to determine. For example, the distance from the optical label to the camera can be determined by the imaging size of the optical label as described above, or the distance from the optical label to the camera can be measured by a binocular camera. In this way, the second imaging position of the optical tag in the second image captured by the second camera can be determined more conveniently or faster with an acceptable error.

In another embodiment, in step 704, the second imaging position may be set to be the same as the first imaging position. In the above, the derivation obtained and, where fx, fy are the focal lengths of the camera in the x and y directions, and dx, dy are the offsets of the two camera coordinate systems in the X and Y directions. fx, fy, dx, dy are usually much smaller than Z (usually a distance of a few meters to tens of meters). Therefore, in some applications that do not require high accuracy, the sum can be considered to be approximately equal to 0, and the second The imaging position is set to be the same as the first imaging position. This method of directly using the imaging position of the light tag in the first image as its imaging position in the second image will bring some errors, but it improves the efficiency and reduces the amount of calculation. Therefore, in some cases, the accuracy requirements are not required. It is very advantageous in high applications. In particular, for the case where the optical tag is recognized at a long distance (Z is large), the error caused by the above method is actually very small, and will not affect the user experience.

The device mentioned in this article may be a device carried by a user (for example, a mobile phone, a tablet computer, smart glasses, a smart helmet, a smart watch, etc.), but it is understood that the device may also be a machine that can move autonomously, for example, UAVs, unmanned vehicles, robots, etc., which are equipped with image acquisition devices, such as cameras.

In an embodiment of the present invention, the present invention can be implemented in the form of a computer program. The computer program can be stored in various storage media (for example, a hard disk, an optical disk, a flash memory, etc.), and when the computer program is executed by a processor, it can be used to implement the method of the present invention.

In another embodiment of the present invention, the present invention can be implemented in the form of an electronic device. The electronic device includes a processor and a memory, and a computer program is stored in the memory. When the computer program is executed by the processor, it can be used to implement the method of the present invention.

References to "various embodiments", "some embodiments", "one embodiment", or "an embodiment" herein refer to the specific features, structures, or properties described in connection with the embodiments included in In at least one embodiment. Therefore, the appearances of the phrases "in various embodiments", "in some embodiments", "in one embodiment", or "in an embodiment" in various places throughout this document do not necessarily refer to the same implementation example. In addition, specific features, structures, or properties can be combined in any suitable manner in one or more embodiments. Therefore, a specific feature, structure, or property shown or described in one embodiment can be combined in whole or in part with the feature, structure, or property of one or more other embodiments without limitation, as long as the combination is not non-limiting. Logical or not working. Expressions similar to "according to A" or "based on A" appearing in this article mean non-exclusive, that is, "according to A" can cover "only according to A" or "according to A and B" unless specifically The statement or the context clearly knows its meaning as "only based on A". The steps described in a certain order in the method flow do not have to be executed in this order. On the contrary, the execution order of some of the steps can be changed, and some steps can be executed concurrently, as long as it does not affect the realization of the solution. In addition, the various elements in the drawings of the present application are for illustrative purposes only, and are not drawn to scale.

Thus, several aspects of at least one embodiment of the present invention have been described, and it can be understood that various changes, modifications and improvements can be easily made by those skilled in the art. Such changes, modifications and improvements are intended to be within the spirit and scope of the present invention. Although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described here, and also includes various changes and changes made without departing from the scope of the present invention.

Claims

A method for presenting information related to an optical communication device, including:

Use the first camera to obtain the first image including the optical communication device in the optical communication device recognition mode;

Obtaining position information of the optical communication device relative to the first camera based on the first image;

Obtaining the position information of the optical communication device relative to the second camera according to the position information of the optical communication device relative to the first camera;

Use a second camera to obtain a second image including the optical communication device; and

The information related to the optical communication device is presented on the second image according to the position information of the optical communication device relative to the second camera.
The method according to claim 1, wherein obtaining the position information of the optical communication device relative to the second camera according to the position information of the optical communication device relative to the first camera comprises:

According to the position information of the optical communication device relative to the first camera, the relative pose information between the first camera and the second camera is used to obtain the position information of the optical communication device relative to the second camera.
The method according to claim 1, wherein obtaining position information of the optical communication device relative to the first camera based on the first image comprises:

The position information of the optical communication device relative to the first camera is obtained based on the imaging of the optical communication device in the first image.
The method according to claim 3, wherein the obtaining the position information of the optical communication device relative to the first camera based on the imaging of the optical communication device in the first image comprises:

Obtaining the distance of the optical communication device relative to the first camera based on the imaging size of the optical communication device in the first image;

Obtaining the direction of the optical communication device relative to the first camera based on the imaging position of the optical communication device in the first image; and

The position information of the optical communication device relative to the first camera is obtained by the distance and direction of the optical communication device relative to the first camera.
The method according to claim 3, wherein the obtaining the position information of the optical communication device relative to the first camera based on the imaging of the optical communication device in the first image comprises:

According to the coordinates of some points on the optical communication device in the optical communication device coordinate system and the imaging positions of these points in the first image, combined with the internal parameter information of the first camera, the relative position of the optical communication device relative to the first image is obtained. Location information of the camera.
The method according to claim 1, further comprising:

Obtaining posture information of the optical communication device relative to the first camera based on the first image;

Obtaining the posture information of the optical communication device relative to the second camera according to the posture information of the optical communication device relative to the first camera and the relative posture information between the first camera and the second camera,

And wherein, the presenting information related to the optical communication device on the second image according to the position information of the optical communication device relative to the second camera includes:

The information related to the optical communication device is presented on the second image according to the position information and posture information of the optical communication device relative to the second camera.
The method according to claim 6, wherein obtaining the posture information of the optical communication device relative to the first camera based on the first image comprises:

Obtain posture information of the optical communication device relative to the first camera based on imaging of the optical communication device in the first image.
The method according to claim 7, wherein said obtaining the posture information of the optical communication device relative to the first camera based on the imaging of the optical communication device in the first image comprises:

Obtain the posture information of the optical communication device relative to the first camera by analyzing the perspective deformation of the imaging of the optical communication device in the first image; or

According to the coordinates of some points on the optical communication device in the optical communication device coordinate system and the imaging positions of these points in the first image, the posture information of the optical communication device relative to the first camera is obtained.
The method according to claim 1, wherein presenting the information related to the optical communication device on the second image according to the position information of the optical communication device relative to the second camera comprises:

According to the position information of the optical communication device relative to the second camera, the imaging position in the second image corresponding to the position information is determined, and the imaging position in the second image is displayed and displayed at the imaging position in the second image. The information about the optical communication device.
A method for presenting information related to an optical communication device, including:

Use the first camera to obtain the first image including the optical communication device in the optical communication device recognition mode;

Obtaining the first imaging position of the optical communication device in the first image;

Use a second camera to obtain a second image including the optical communication device; and

Presenting information related to the optical communication device at a second imaging position in the second image according to the first imaging position,

Wherein, the first camera and the second camera are installed on the same plane and have the same posture and internal parameters.
The method of claim 10, wherein:

The second imaging position is the same as the first imaging position.
The method of claim 10, wherein:

According to the relative offset between the first camera and the second camera, the first imaging position, the Z coordinate of the optical communication device in the first camera coordinate system or the second camera coordinate system, or the optical communication device The vertical distance between the installation plane of the first camera and the second camera is used to determine the second imaging position.
The method of claim 10, wherein:

The second imaging position is determined according to the relative offset between the first camera and the second camera, the first imaging position, and the distance from the optical communication device to the first camera or the second camera.
A computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it can be used to implement the method of any one of claims 1-13.
An electronic device comprising a processor and a memory, and a computer program is stored in the memory, and when the computer program is executed by the processor, it can be used to implement the method according to any one of claims 1-13.
A computer program product, when the computer program product is executed by a processor, it can be used to implement the method of any one of claims 1-13.